# Build systems for the web and node.js / io.js
Current javascript state of the art compared to sigh, a functional reactive declarative build automation system.
Wikipedia: "In computing, a pipeline is a set of data processing elements connected in series, where the output of one element is the input of the next one."
* How to build code:
* Task runner
* Asset pipeline
* Good for anything that reacts to code changes
* Transpile es6/coffee/typescript/scss.
* Run tests after everything is built.
* Create production version of code (minified etc.)
* Talks to your browser via livereload.
* Incrementally rebuild.
* Reactive development automation system
# History
* "All your base are belong to us" - 1998
* "Don't tase me bro" - 2007
* node 0.0.1 - May 2009.
* CoffeeScript 0.1 - December 2009.
* npm 0.0.1 - January 2010 (node at 0.1).
* jake 0.1.5 - August 2010.
* npm 1.0.1 - April 2011 (node at 0.4)
* grunt 0.1.0 - January 2012. (& "Grumpy Grandma")
* gulp 0.0.1 - July 2013, 1.0.0 - September (& "Grumpy Cat").
* traceur 0.0.10 - January 2014.
* broccoli (first beta) - Feb 2014.
# Before grunt
* Gnu make: Makefile (gruntfile.js, gulpfile.js etc.)
* jake
* Shell scripts
* Custom build scripts written in javascript
* Other stuff
Makefile syntax
❯ make
❯ touch
❯ make
:
@echo feeding lovely sad cat
feed --animal $^
:Nobody INTERFERES
with Mr T or Mr T's
BINS SUCKAH
cat $^ > $@
:
@mplayer inspirational-montage.mkv
Hey SUCKAH.
Mr T 'aint WEARING no Red HAT SUCKAH.
And don't you COME in to MY GARAGE
at NIGHT and KICK over MY BINS.
## Jake
* Jake could do pretty much everything make could.
* Much more powerful/flexible task scheduling than grunt.
* Never caught on, maybe due to lack of plugin system?
## Can schedule tasks to run after current task
```javascript
// i cannot ride the catbus like this!!!
grunt.registerTask('ride-catbus', function() {
var done = this.async()
grunt.task.run(['wait-at-bus-stop', 'give-totoro-umbrella'])
console.log('Where are you totoro?')
setTimeout(function() {
console.log('Totoro... Did I do something wrong???')
done()
// after this function exits the tasks are run
}, 10000)
})
```
Not before or during current task. ~~That would be wrong.~~
## A task can schedule other tasks in sequence
```javascript
// i just need to write another task to ride the catbus
grunt.registerTask('get-on-catbus', function() {
// ...
})
grunt.registerTask('ride-catbus', [
'wait-at-bus-stop',
'give-totoro-umbrella',
'other-stuff-happens',
'get-on-catbus'
])
```
* No release for 11 months.
* Is a task runner rather than a build system.
* It was just used as one.
* Because nobody seemed to like jake.
* And there was nothing else.
* Pipeline = plugins taking turns to compile code in temp directories.
* yeoman angular fullstack generator "best practice" gruntfile, 667 lines long:
# Gulp
* gulp was released 1.5yrs after grunt in July 2013
* pipelines are node streams
* plugins are (mostly) stream transforms, 1 source file in 1 destination file out.
* The streams are exposed directly to build files.
```javascript
// gulpfile.js
var gulp = require('gulp')
var traceur = require('gulp-traceur')
var sourcemaps = require('gulp-sourcemaps')
var concat = require('gulp-concat')
var merge = require('merge-stream')
var paths = {
es6: ['app/**/*.js'],
raw: ['raw/*.js']
}
gulp.task('client-scripts', function() {
merge(
gulp.src(paths.es6)
.pipe(sourcemaps.init())
.pipe(traceur()),
gulp.src(paths.raw)
.pipe(sourcemaps.init())
)
.pipe(concat('output.js'))
.pipe(sourcemaps.write())
.pipe(gulp.dest('dist/assets'))
})
gulp.task('watch', function() {
gulp.watch(paths.es6.concat(paths.raw), ['client-scripts'])
})
gulp.task('default', ['client-scripts', 'watch'])
```
* ... that naive gulpfile is not good:
* No incremental rebuilds.
* Invalid source map.
* Need to restart gulp on error.
* Concatenating source maps doesn't always work.
* Streams transforms are built around transforming files 1:1
* n:n/1:n/n:1 transforms more difficult.
* Node stream API not so good for this task:
* Node streams stop accepting incoming data on first error.
* gulp-plumber plugin needed to monkey patch it.
```javascript
// gulpfile.js
var gulp = require('gulp')
var traceur = require('gulp-traceur')
var sourcemaps = require('gulp-sourcemaps')
var concat = require('gulp-concat')
var merge = require('merge-stream')
// these are needed for incremental rebuilds
var cache = require('gulp-cached')
var remember = require('gulp-remember')
var order = require('gulp-order')
// this is needed to fix bad stream behaviour
var plumber = require('gulp-plumber')
var paths = {
es6: ['app/**/*.js'],
raw: ['raw/*.js']
}
var allPaths = paths.es6.concat(paths.raw)
gulp.task('client-scripts', function() {
merge(
gulp.src(paths.es6)
.pipe(sourcemaps.init())
.pipe(plumber())
.pipe(cache('traceurCode'))
.pipe(traceur()),
gulp.src(paths.raw)
.pipe(sourcemaps.init())
)
.pipe(remember('traceurCode'))
.pipe(order(allPaths))
.pipe(concat('output.js'))
.pipe(sourcemaps.write())
.pipe(gulp.dest('dist/assets'))
})
gulp.task('watch', function() {
gulp.watch(allPaths, ['client-scripts'])
})
gulp.task('default', ['client-scripts', 'watch'])
```
Mr T recommends you EAT your BROCCOLI SUCKAH.
Mr T also RECOMMENDS you stay AWAY
from Mr T's BINS and Mr T's VAN.
# Broccoli
* First release May 2013
* Currently in beta
* Plugins take one or more trees (filesystem directories).
* Output one or more trees (containing built assets).
* "The filesystem is the API"
# That example from earlier in broccoli
```javascript
var funnel = require('broccoli-funnel')
var mergeTrees = require('broccoli-merge-trees')
var es6Transpiler = require('broccoli-babel-transpiler')
var concat = require('broccoli-concat')
var es6Tree = funnel('app', {
destDir: 'es6',
include: [ /\.js$/ ]
})
var es6TranspiledTree = esTranspiler(es6Tree)
var rawJsTree = funnel('raw', {
destDir: 'raw',
include: [ /\.js$/ ]
})
var allJsTree = mergeTrees([ es6TranspiledTree, rawJsTree ])
var concatTree = concat(allJsTree, {
// ensure raw files concatted after es6 files
inputFiles: [ 'es6/*', 'raw/*' ],
outputFile: 'dist/assets/output.js'
})
module.exports = concatTree
```
# But...
* Source maps rarely working.
* Not working in previous example.
* Many plugins lack transitive application.
* Use of filesystem questionable:
* Plugins must clean-up.
* Infrastructure code more complicated, temp files etc.
* Filesystem is slow.
* "No parallelism"
* "Only a 40% speed gain."
* Possible but "broccoli's primitives + helper code encourage deterministic sequential code patterns."
* Brocfile.js files can get verbose.
# jspm
* Based on EcmaScript 6 module loader.
* Gives you everything browserify + bower does and more.
* `jspm install bower:lodash`
* `jspm install github:bluebird`
* `jspm install npm:totoro-love-helper`
* Can load AMD/CommonJS and EcmaScript 6 modules.
* Can load/transpile dependencies at **run-time**.
* ... or can prepare self-contained bundles.
* Have/Get to use ES6 modules + ES6 transpiler in your code:
```javascript
import _ from 'lodash'
import { promisify } from 'bluebird'
```
* In development mode modules will be transpiled many times:
* Each time browser reloads.
* During tests (once per browser + reload).
* Modules are transpiled one by one:
* Delaying page load.
* High CPU usage.
* Janking main JavaScript thread.
* "Bundle" builds slow, not incrementally rebuilt.
* Can use jspm + sigh together for this.
# Other build systems
* metalsmith
* gobble
* brunch
## Those without changes in 6+ months.
* plumber
* cha
* grunt
# Problems with most things
* You make a change while the tests are running and:
* Change is ignored.
* Waits until tests have finished then runs them again.
* Production source maps.
* Need special handling for watching files.
# sigh
* Functional Reactive Programming
* Like lodash of streams (flatten, reduce etc.)
* `n:n` / `1:n` / `1:1` operations: stream payload is array of events.
* Can switch between sequential/parallel easily.
* Does not use fs except when absolutely necessary.
* Provides lovely APIs for plugin writers
* Working with processes.
* Working with source maps (application, concatenation).
* Source maps from minified/transpiled files back to original sources.
* Very neat Assetfile.js syntax.
* Plugin authors have powerful FRP stream API.
* Stream API hidden from users by pipeline compiler.
# That example from earlier with sigh
```javascript
// sigh.js
var glob, babel, concat // plugins injected into globals
module.exports = function(pipelines) {
pipeline.build = [
glob('app/**/*.js'),
babel(),
glob('raw/*.js'),
concat('dist/assets/output.js')
]
}
```
# A more complicated example
```javascript
// sigh.js
var glob, babel, sass, write, livereload, process
module.exports = function(pipelines) {
pipelines.client = [
merge(
[
glob({ basePath: 'src' }, '*.js', 'app.scss'),
sass(),
babel(),
write({ clobber: '!(jspm_packages)' }, 'build'),
],
glob('index.html', 'config.js')
),
livereload()
]
pipelines.server = [
glob('server.js'),
process('node server.js')
]
}
```
# With more parallelism and environments
```javascript
var glob, babel, sass, merge, write, livereload, process
var env, concat, uglify
module.exports = function(pipelines) {
var globOpts = { basePath: 'src' }
pipelines.client = [
merge(
[
merge(
[ glob(globOpts, 'app.scss'), sass() ],
[
glob(glopOpts, '*.js'),
babel(),
env(
[ concat('combined.js'), uglify() ],
[ 'production', 'staging' ]
)
]
),
write({ clobber: '!(jspm_packages)' }, 'build'),
],
glob('index.html', 'config.js')
),
livereload()
]
pipelines.server = [
glob('server.js'),
process('node server.js')
]
}
```
# Connecting pipelines together
```javascript
var glob, pipeline, babel, debounce, write, mocha
module.exports = function(pipelines) {
pipelines['build-sources'] = [
glob({ basePath: 'src' }, '*.js', 'plugin/*.js'),
babel({ modules: 'common' }),
write('lib')
]
pipelines['build-tests'] = [
glob({ basePath: 'src/test' }, '*.js', 'plugin/*.js'),
babel({ modules: 'common' }),
write('lib/test')
]
pipelines.alias.build = ['build-sources', 'build-tests']
pipelines['run-tests'] = [
// pipeline does not force pipelines to run, if user
// runs 'sigh build-tests run-tests' then
// 'build-source' will not be run
pipeline('build-sources', 'build-tests'),
debounce(700),
// { activate: true } forces mocha pipeline to
// run even if it was not selected
pipeline({ activate: true }, 'mocha')
]
// explicit pipelines only run if selected or activated.
pipelines.explicit.mocha = [
mocha({ files: 'lib/test/**/*.spec.js' })
]
}
```
# sigh uses multiple processes for extra speed.
* Tests: keeps two processes cached and waiting.
* Tests killed when new value received while running.
* Second process already waiting to avoid startup delay.
* Safe-parallelism:
* Array of events split up and distributed to processes.
* Collects results from all processes.
* Forwards all events in one payload, just as if one process did it.
* Compiles ES6 files faster than the native transpiler.
# Structure
* Make bigger pipelines by connecting pipelines together.
* Supports gulp plugins.
# sigh helps scaffold plugins (with yeoman)
❯ sigh -p plugin-name
? What is the name of this plugin? plugin name
? Which github username/organisation should own this plugin? username
? What's your github author? Your Name <name@email.net>
? Which of the following apply to your plugin? (Press <space> to select)
❯◉ Maps input files to output files 1:1
◯ Spreads work over multiple CPUs
? Which features would you like to use? (Press <space> to select)
❯◉ CircleCI integration
◯ TravisCI integration
? Which dependencies do you need? (Press <space> to select)
❯◉ bluebird
◉ lodash
◉ sigh-core
Github organisation/author and CI remembered for next time.
Then fill in the remaining code and publish
❯ cd plugin-name ❯ sigh -w
# edit code, sigh will recompile and run tests as you do ❯ npm publish
FIN
* # Maryam
* Mr. T
* [http://bubblegun.com](http://bubblegun.com)
* [http://www.yellow5.com/pokey/](Pokey the penguin)
* Storz-Bickel
@echo feeding lovely sad cat
feed --animal $^
cat $^ > $@
@mplayer inspirational-montage.mkv
