Streamlining your streaming build system

Gulp

Gulp is pretty slick, but it doesn't stop you from writing pretty chunky gulpfiles, and deploying it consistently can be tricky.

A few of ways I keep my builds in check include:

If you want to skip to a finished example of my set up, check out:

https://github.com/stevelacey/gulp-example

Gulping with Coffee

If you're into CoffeeScript, simply register the compiler in your gulpfile.js, and require some coffee, it's not for everyone, but I think it's great for Gulp.

gulpfile.js

require('coffee-script/register');  
require('./gulpfile.coffee');  

gulpfile.coffee

g = require("gulp")

g.task "build", ["clean"], -> g.start "scripts", "styles"  

Filing away tasks

Although Gulp syntax is pretty concise compared to a lot of the alternatives, I still found that the size and number of tasks in my gulpfile was growing pretty quick.

A quick and easy solution is to move your tasks into individual files, say, in a /gulp directory, and load them in, which is simple enough:

glob = require("glob")

require file for file in glob.sync "./gulp/*.coffee"  

However, doing this raises a few issues:

  • You're probably sharing a number of dependencies between yours
  • You're probably not too keen on re-requiring everything everywhere

Which leads us on to...

Dynamically requiring plugins

As you'll see in gulp-example, I'm installing 19 gulp-related dependencies, requiring all of which is a massive pain, especially across multiple task files.

Gulp load plugins allows you to require in packages based on a prefix, e.g. gulp-*, tidy.

You're already going to be requiring gulp in your task files, you might want to consider monkey patching them onto it:

gulpfile.coffee

g = require("gulp")  
g.p = require("gulp-load-plugins")()  

Now you can write task files like this:

gulp/styles.coffee

g = require("gulp")

g.task "styles", ->  
  g.src "styles/*.scss"
    .pipe g.p.sass style: "expanded", errLogToConsole: g.e == 'dev'
    .pipe g.css()
    .pipe g.p.autoprefixer "last 2 version", "safari 5", "ie 8", "ie 9", "opera 12.1", "ios 6", "android 4"
    .pipe g.p.concat "main.css"
    .pipe g.dest "web/css"
    .pipe g.reload()

Slick!

Lazypiping common flows

Lazypipe allows you to create an immutable, lazily-initialized pipeline. It's designed to be used in an environment where you want to reuse partial pipelines, a good example being for multiple similar script and style tasks.

gulpfile.coffee

lazy = require("lazypipe")

g.css = lazy()  
  .pipe g.p.autoprefixer "last 2 version" 
  .pipe g.p.cssUrlAdjuster, prepend: "/my/rev/hash/"

gulp/styles.coffee

g.task "styles", ->  
  g.src "styles/*.scss"
    .pipe g.css()
    # ...

gulp/vendor.coffee

g.task "vendor-styles", ["bower"], ->  
  g.src "web/vendor/**/*.css"
    .pipe g.css()
    # ...

Shrinkwrapping for production

Not knowing precisely which versions of packages you're depending on can become pretty troublesome when you're rolling a JS-based build process, especially when working in a team or deploying to multiple environments.

Shrinkwrap solves most of the problems, simply shrinkwrap your project:

npm shrinkwrap  

And it'll create you a npm-shrinkwrap.json, your lockfile, commit it!

Next time, as per usual:

npm install  

As long as an npm-shrinkwrap.json is found, npm install will use the lockfile, and you'll end up with precisely the same versions of the packages you require, as well as their dependencies. Sorted.

Deploying with Capistrano 3

Adding NPM and Bower dependencies into your project add a number of challenges for deployment.

Sure, if you're deploying to a known, single-tenant environment, you can probably just install everything globally and stop caring, but if you're not, what are your options when it comes to hosting multiple projects with common dependencies at multiple versions?

I tackle this problem by modifying $PATH, so that as far as the build process knows, these global-only dependencies ARE installed globally, when really, they're not.

Capistrano 3 is the perfect tool for doing this, as it simplifies adjusting env vars on a per task basis:

namespace :npm do  
  task :install do
    on roles :app do
      within release_path do
        execute :npm, :install
      end
    end
  end
end

namespace :bower do  
  task :install do
    on roles :app do
      within release_path do
        with path: "#{release_path}/node_modules/.bin:$PATH" do
          execute :bower, :update, "--config.interactive=false"
        end
      end
    end
  end
end

namespace :gulp do  
  task :build do
    on roles :app do
      within release_path do
        with path: "#{release_path}/node_modules/.bin:$PATH" do
          execute :gulp, :build
        end
      end
    end
  end
end

after "deploy:updating", "npm:install"  
after "deploy:updating", "bower:install"  
after "deploy:updating", "gulp:build"  

What this does, is allow for Capistrano to perform a deploy that includes a bower install and a gulp build, disregarding the fact that bower, gulp nor even CoffeeScript, are installed on the server. All that is required to allow this, is that these otherwise-global dependencies are described in the package.json, so that they are installed into ./node_modules.

"dependencies": {
  "bower": "~1.3.1",
  "coffee-script": "~1.7.1",
  "gulp": "~3.6.0",
  ...
}

Simple, self-contained, shipped!

Summary

So thanks for reading this far! I hope some of my solutions prove useful, do take heed that I by no means present these as a best practice, they're simply how I get stuff done.

I'd love to hear alternative solutions to any or all of the above in the comments below, or catch me on Twitter.

If you want to see a finished example of my set up, check out: https://github.com/stevelacey/gulp-example