Unlocking esbuild’s Fast JavaScript API: Cross‑Language IPC Explained

Topic Overview

esbuild is a bundler implemented in Go, known for its simple configuration and lightning‑fast compilation speed, and it is widely adopted. It provides a JavaScript API that is fully compatible with the JavaScript ecosystem.

The following example is taken from the esbuild documentation.

import { transformSync } from 'esbuild';

transformSync('let x: number = 1', {
  loader: 'ts',
});

At this point you may wonder: esbuild is written in Go, yet we can call it like a normal npm package in JavaScript—how does it achieve cross‑language invocation? This article investigates that question.

Inter‑Process Communication

Intuitively, inter‑process communication (IPC) can realize this effect. A simple example is used to verify the hypothesis.

vite, a JavaScript‑based build tool that depends on esbuild, is created and started with yarn run dev. The ps command finds the current process ID, and pstree visualizes the process hierarchy.

-+= 97048 user node .../yarn.js run dev
 \-+- 97055 user node .../.bin/vite
   \--- 97059 user .../esbuild --service=0.13.15 --ping

The output of pstree clearly shows the relationship between processes, confirming the earlier guess.

Initial Implementation

esbuild CLI offers two usage patterns:

# 1
cat src/main.ts | esbuild --loader=ts

# 2
esbuild src/main.ts

After executing the command, the compiled code is printed to the console. The above shell commands can be transformed into equivalent Node.js code.

import { spawn } from 'child_process';

const esbuild = spawn('./node_modules/esbuild-darwin-64/bin/esbuild', [
  'package.json',
]);

esbuild.stdout.on('data', (chunk) => {
  console.log(chunk.toString());
});

The current implementation has obvious drawbacks:

Only one file can be compiled at a time.

Compiling multiple files creates processes repeatedly.

Improvement

A better approach is to keep the esbuild process running after it starts, writing data to its stdin while decoding data from its stdout.

Note that esbuild is started with the following command: esbuild --service=0.13.15 --ping The --ping flag makes the esbuild process stay alive instead of exiting, allowing it to continuously accept requests via stdin.

Communication Protocol

Model Construction

Using the process’s stdin/stdout enables communication. On top of that, a Packet structure is defined to describe the transmitted data:

export interface Packet {
  id: number;
  isRequest: boolean; // usually true
  value: Value;
}

The value field represents the protocol payload. For example, TransformRequest denotes a file‑transform request, while TransformResponse represents the corresponding response.

type Value = TransformRequest | TransformResponse | Others;

export interface TransformRequest {
  command: 'transform';
  flags: string[];
  input: string;
  // ...
}

export interface TransformResponse {
  code: string;
  // ...
}

Serialization and Deserialization

Serialization converts an object’s state into a storable format; deserialization reconstructs the object from that format. esbuild’s messages use a fixed‑length encoding where the first four bytes indicate the message body length, followed by the body.

+---+---+---+---+---+---+---+---+---+----+
| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | .. |
+---+---+---+---+---+---+---+---+---+----+
|   length   |        body          |
+------------+----------------------+

Data types are categorized as:

Fixed‑length: boolean, uint32, null

Variable‑length: array, string, object

Byte order is little‑endian. The example below illustrates this:

const buffer = Buffer.from([0x01, 0x02, 0x03, 0x04]);
buffer.readUInt32LE(); // = 67305985
buffer.readUInt32BE(); // = 16909060 = 0x01020304

Serializing the string abc yields:

+-----+----+----+----+----+----+----+----+
|  1  | 2  | 3  | 4  | 5  | 6  | 7  | 8  |
+-----+----+----+----+----+----+----+----+
| 03  | 03 | 00 | 00 | 00 | 61 | 62 | 63 |
+-----+----+----+----+----+----+----+----+
| str | length: 3 | a  | b  | c  |
+-----+-------------------+----+----+----+

The whole process is summarized in the diagram below.

Implementation

After the principles are explained, we start writing code.

Codec

Before implementing the codec, we create an auto‑expanding buffer to abstract read/write details.

export class PacketBuffer {
  private buffer: Buffer;

  scale() {}

  writeUint32() {}

  readUint32() {}
}

Then we implement encode and decode methods.

export function encode(packet: Packet): Buffer {
  const buffer = new PacketBuffer();
  buffer.writeUint32(id);
  // ...
  return buffer.slice();
}

export function decode(buffer: PacketBuffer): Packet {
  // ...
  return packet;
}

Stream Read/Write

During IPC, data is stream‑oriented. Packets are encoded before being written to the stream, and streams are decoded back into packets. Writing is straightforward; reading uses two transform streams: FixedLengthTransform splits the stream by length, then PacketTransform converts the byte chunks into Packet objects.

import { pipeline } from 'stream';

// Split stream by packet length
export class FixedLengthTransform extends Transform {}

// Convert stream chunks to Packet objects
export class PacketTransform extends Transform {}

const transform = pipeline(
  esbuild.stdout,
  new FixedLengthTransform(),
  new PacketTransform(),
  handlerError(),
);

transform.on('data', (packet) => {});

API Wrapper

By assembling the above components, we expose a JavaScript API similar to the official esbuild npm package. Users can call the provided functions as needed.

export class API {
  transformTs(code: string) {}

  transformJson(code: string) {}

  transform(code: string, option: Option) {}
}

RPC

The cross‑process (cross‑language) communication described can be viewed as a minimal Remote Procedure Call (RPC) system without network transport. For front‑end developers, this illustrates the core concept of RPC.

Conclusion

In the front‑end toolchain landscape, bundlers built with Go or Rust are emerging rapidly. This article uses the esbuild JavaScript API as a case study to analyze how cross‑language APIs are implemented, helping readers understand bundler internals.

It also delves into the design of esbuild’s binary protocol and its encoding/decoding implementation, enabling readers to grasp Node process and stream concepts, design custom protocols, and better understand related RPC concepts.