How HappyPack Supercharges Webpack Builds with Multi‑Process Parallelism

HappyPack is a webpack plugin that accelerates code compilation by using a multi‑process model. After deploying it on a 16‑core build server, significant build‑time reductions were observed.

Webpack Loading Configuration

var HappyPack = require('happypack');
var happyThreadPool = HappyPack.ThreadPool({ size: os.cpus().length });
// ...other configuration
module: {
  loaders: [
    {
      test: /\.less$/,
      loader: ExtractTextPlugin.extract(
        'style',
        path.resolve(__dirname, './node_modules', 'happypack/loader') + '?id=less'
      )
    }
  ]
},
plugins: [
  new HappyPack({
    id: 'less',
    loaders: ['css!less'],
    threadPool: happyThreadPool,
    cache: true,
    verbose: true
  })
]

The example shows only the part of the configuration that extracts HappyPack settings. Like other loaders (e.g., extract-text-webpack-plugin), HappyPack works through the interaction of webpack loader and plugin components.

HappyPack File Parsing

HappyPlugin.js

In webpack, a plugin runs throughout the entire compilation process. For HappyPack, the first step is the plugin initialization, where core parameters are set.

1. Basic Parameter Setup

function HappyPlugin(userConfig) {
  if (!(this instanceof HappyPlugin)) {
    return new HappyPlugin(userConfig);
  }
  this.id = String(userConfig.id || ++uid);
  this.name = 'HappyPack';
  this.state = {
    started: false,
    loaders: [],
    baseLoaderRequest: '',
    foregroundWorker: null,
  };
  // ...other config omitted
}

The two identifiers are: id: the ID defined in the configuration, used to match loaders with the plugin. name: always HappyPack, allowing quick identification of the corresponding plugin in loader code.

These identifiers are later referenced in loader code such as:

function isHappy(id) {
  return function(plugin) {
    return plugin.name === 'HappyPack' && plugin.id === id;
  };
}

2. Thread‑Pool Initialization

function HappyPlugin(userConfig) {
  // ...basic setup omitted
  this.threadPool = this.config.threadPool || HappyThreadPool({
    id: this.id,
    size: this.config.threads,
    verbose: this.config.verbose,
    debug: this.config.debug,
  });
  // ...cache init omitted
}

The thread pool actually manages child processes (referred to as “processes” rather than threads). Whether the configuration uses threads or threadPool, a HappyThreadPool object is created.

2.1 HappyThreadPool.js

function HappyThreadPool(config) {
  var happyRPCHandler = new HappyRPCHandler();
  var threads = createThreads(config.size, happyRPCHandler, {
    id: config.id,
    verbose: config.verbose,
    debug: config.debug,
  });
  // ...return object omitted
}

The pool creates child processes via createThreads and wraps each in a HappyThread object.

2.1.1 HappyRPCHandler.js

function HappyRPCHandler() {
  this.activeLoaders = {};
  this.activeCompiler = null;
}

The handler binds the current loader and compiler, exposing RPC methods for both.

Compiler RPCs include a resolve method that forwards to webpack's resolver.

Loader RPCs provide emitWarning, emitError, addDependency, and addContextDependency functions.

2.1.2 HappyThread.js (Child Process Management)

HappyThread = {
  open: function(onReady) {
    fd = fork(WORKER_BIN, [id], { execArgv: [] });
    // ...message handling omitted
  },
  configure: function(compilerOptions, done) {
    // ...implementation omitted
  },
  compile: function(params, done) {
    // ...implementation omitted
  },
  isOpen: function() { return !!fd; },
  close: function() {
    fd.kill('SIGINT');
    fd = null;
  }
};

The open method forks the worker script ( HappyWorkerChannel.js) and establishes IPC communication.

2.1.2.1 HappyWorkerChannel.js

var HappyWorker = require('./HappyWorker');
if (process.argv[1] === __filename) {
  startAsWorker();
}
function startAsWorker() {
  HappyWorkerChannel(String(process.argv[2]), process);
}
function HappyWorkerChannel(id, stream) {
  var worker = new HappyWorker({ compiler: fakeCompiler });
  stream.on('message', accept);
  stream.send({ name: 'READY' });
  function accept(message) {
    // ...message routing omitted
  }
}

The worker receives messages such as COMPILE, performs the compilation, writes the result to disk, and replies with a COMPILED message.

3. Compilation Cache Initialization

this.cache = HappyFSCache({
  id: this.id,
  path: this.config.cachePath ?
    path.resolve(this.config.cachePath.replace(/\[id\]/g, this.id)) :
    path.resolve(this.config.tempDir, 'cache--' + this.id + '.json'),
  verbose: this.config.verbose,
  generateSignature: this.config.cacheSignatureGenerator
});
HappyUtils.mkdirSync(this.config.tempDir);

The cache stores compiled files in a temporary directory and maps source files to their compiled counterparts.

3.1 HappyFSCache.js

exports.load = function(currentContext) { /* ... */ };
exports.save = function() { /* ... */ };
exports.getCompiledSourceCodePath = function(filePath) {
  return cache.mtimes[filePath] && cache.mtimes[filePath].compiledPath;
};
exports.updateMTimeFor = function(filePath, compiledPath, error) {
  cache.mtimes[filePath] = {
    mtime: generateSignature(filePath),
    compiledPath: compiledPath,
    error: error
  };
};
// ...other methods omitted

During compilation, getCompiledSourceCodePath retrieves cached output, while updateMTimeFor records new cache entries.

HappyLoader.js

function HappyLoader(sourceCode, sourceMap) {
  var happyPlugin, happyRPCHandler;
  var callback = this.async();
  var id = getId(this.query);

  happyPlugin = this.options.plugins.filter(isHappy(id))[0];

  happyPlugin.compile({
    remoteLoaderId: remoteLoaderId,
    sourceCode: sourceCode,
    sourceMap: sourceMap,
    useSourceMap: this._module.useSourceMap,
    context: this.context,
    request: happyPlugin.generateRequest(this.resource),
    resource: this.resource,
    resourcePath: this.resourcePath,
    resourceQuery: this.resourceQuery,
    target: this.target,
  }, function(err, outSourceCode, outSourceMap) {
    callback(null, outSourceCode, outSourceMap);
  });
}

The loader extracts the id from its query, finds the matching HappyPack plugin, and forwards the compilation request to the plugin.

Actual Execution Flow

When webpack finishes its basic configuration, it triggers the run event. HappyPack registers a listener on this event:

HappyPlugin.prototype.apply = function(compiler) {
  compiler.plugin('run', this.start.bind(this));
};

The start method runs a series of asynchronous steps:

HappyPlugin.prototype.start = function(compiler, done) {
  async.series([
    function registerCompilerForRPCs(callback) { /* ... */ },
    function normalizeLoaders(callback) { /* ... */ },
    function resolveLoaders(callback) { /* ... */ },
    function loadCache(callback) { /* ... */ },
    function launchAndConfigureThreads(callback) { /* ... */ },
    function markStarted(callback) { /* ... */ }
  ], done);
};

Key steps include:

registerCompilerForRPCs : binds the compiler to the RPC handler.

normalizeLoaders : converts loader strings (e.g., css!less) into an array of loader objects.

resolveLoaders : uses webpack's resolver to locate each loader on disk and builds the final loader request string.

loadCache : loads previous build caches if caching is enabled.

launchAndConfigureThreads : starts the child‑process pool and configures each thread.

Thread launching is performed by HappyThreadPool.start, which opens all threads that are not already open:

HappyThreadPool.prototype.start = function(done) {
  async.parallel(
    threads.filter(not(send('isOpen'))).map(get('open')),
    done
  );
};

Utility functions send, not, and get are simple higher‑order helpers that filter, retrieve, and invoke methods on thread objects.

Overall, HappyPack integrates tightly with webpack’s plugin and loader system, creates a pool of worker processes, caches compiled results, and orchestrates the entire build pipeline to achieve substantial speed‑ups.