Mastering Node.js Buffer: From Basics to Performance Optimization

Quick Navigation

Buffer Introduction

What is binary data?

What is Stream?

What is Buffer?

Buffer Basic Usage

Creating Buffer

Buffer character encoding

String ↔ Buffer conversion

Buffer Memory Mechanism

Memory allocation principle

8KB limit

Object allocation summary

Buffer Application Scenarios

I/O operations

zlib.js

Encryption/Decryption

Buffer VS Cache

Buffer VS String

Buffer Introduction

Before TypedArray, JavaScript lacked a mechanism to read or manipulate binary data streams. Node.js introduced the Buffer class as part of its API to interact with octet streams in TCP, file systems, and other contexts. In short, Node.js can handle binary stream data.

Buffer is used for binary data operations without requiring require. It is essential for network protocols, databases, image processing, and file I/O. When creating a Buffer, its size is fixed and cannot be resized.

What is binary data?

Binary data uses 0 and 1 to represent information. For example, the decimal number 66 becomes 01000010 in binary.

What is Stream?

A Stream abstracts input/output devices such as files, networks, or memory. Data flows directionally: an input stream reads data from a source, while an output stream writes data to a destination. Streams allow large files to be processed piece by piece, like water flowing through a pipe.

What is Buffer?

Buffer acts as a waiting area for data. When data arrives faster than it can be processed, excess data is stored in the buffer until the program can handle it. This concept is illustrated with a bus stop analogy where passengers wait for the next bus.

Buffer Basic Usage

Below are common Buffer APIs. For full details, refer to the Node.js documentation.

Creating Buffer

Prior to Node.js 6.0.0, Buffers were created with the constructor new Buffer(). Modern code should use:

const b1 = Buffer.from('10');
const b2 = Buffer.from('10', 'utf8');
const b3 = Buffer.from([10]);
const b4 = Buffer.from(b3);
console.log(b1, b2, b3, b4); // <Buffer 31 30> <Buffer 31 30> <Buffer 0a> <Buffer 0a>

Buffer.from()

const b1 = Buffer.from('10');
const b2 = Buffer.from('10', 'utf8');
const b3 = Buffer.from([10]);
const b4 = Buffer.from(b3);

Buffer.alloc()

Creates an initialized Buffer that never contains old data.

const bAlloc1 = Buffer.alloc(10); // 10 zero‑filled bytes
console.log(bAlloc1);

Buffer.allocUnsafe()

Creates an uninitialized Buffer; the memory may contain old sensitive data.

const bAllocUnsafe1 = Buffer.allocUnsafe(10);
console.log(bAllocUnsafe1);

Buffer Character Encoding

Supported encodings include:

'ascii' – 7‑bit ASCII

'utf8' – Unicode UTF‑8

'utf16le' – Little‑endian UTF‑16

'ucs2' – Alias of 'utf16le'

'base64' – Base64 encoding

'latin1' – Single‑byte Latin‑1

'binary' – Alias of 'latin1'

'hex' – Hexadecimal representation

const buf = Buffer.from('hello world', 'ascii');
console.log(buf.toString('hex')); // 68656c6c6f20776f726c64

String ↔ Buffer Conversion

String to Buffer

const buf = Buffer.from('Node.js 技术栈', 'UTF-8');
console.log(buf);

Buffer to String

const str = buf.toString('UTF-8', 0, 9); // partial conversion
console.log(str); // Node.js �

Buffer Memory Mechanism

Buffer memory is allocated outside V8's heap (off‑heap) via C++ to avoid frequent system calls. Node.js uses a slab allocation strategy with a default pool size of 8 KB.

Memory Allocation Principle

Node.js pre‑allocates an 8 KB slab. Small Buffers are carved from this slab; large Buffers are allocated directly via C++.

// Simplified allocate(size) logic
function allocate(size) {
  if (size <= 0) return new FastBuffer();
  if (size <= Buffer.poolSize >>> 1) {
    // use slab
    if (size > Buffer.poolSize - poolOffset) createPool();
    const b = new FastBuffer(allocPool, poolOffset, size);
    poolOffset += size;
    alignPool();
    return b;
  } else {
    // large allocation
    return createUnsafeBuffer(size);
  }
}

Key points:

On first load, an 8 KB slab is created.

Buffers ≤ 8 KB are allocated from the slab.

If the slab lacks space, a new slab is created.

Buffers > 8 KB bypass the slab and are allocated directly.

All memory is eventually reclaimed by V8's garbage collector.

Buffer Application Scenarios

Common use cases include:

I/O Operations

const fs = require('fs');
const input = fs.createReadStream('input.txt');
const output = fs.createWriteStream('output.txt');
input.pipe(output);

Streams automatically manage internal buffers.

zlib.js

Node's core zlib module uses Buffers for compression and decompression.

Encryption/Decryption

Buffers are used for keys and IVs in crypto operations.

const crypto = require('crypto');
function handleKey(key) {
  const bytes = Buffer.alloc(16);
  bytes.fill(key, 0, 10);
  return bytes;
}
const [key, iv, algorithm, encoding, cipherEncoding] = ['a123456789', '', 'aes-128-ecb', 'utf8', 'base64'];
let cipher = crypto.createCipheriv(algorithm, handleKey(key), iv);
let crypted = cipher.update('Node.js 技术栈', encoding, cipherEncoding);
crypted += cipher.final(cipherEncoding);
console.log(crypted);

Buffer VS Cache

What is the difference between Buffer and Cache?

Buffer temporarily stores binary stream data during processing (e.g., video buffering).

Cache permanently stores frequently accessed data to speed up retrieval (e.g., in‑memory caches like Redis).

Buffer VS String

A performance test compares sending raw strings versus pre‑converted Buffers over HTTP.

// Server handling /buffer and /string routes
const http = require('http');
let s = '';
for (let i = 0; i < 1024 * 10; i++) s += 'a';
const bufStr = Buffer.from(s);
const server = http.createServer((req, res) => {
  if (req.url === '/buffer') {
    res.end(bufStr);
  } else if (req.url === '/string') {
    res.end(s);
  }
});
server.listen(3000);

Benchmark results (using ab):

String endpoint: 21 815 requests, 363.58 req/s, 3 662 KB/s.

Buffer endpoint: 50 000 requests, 907.24 req/s, 9 138 KB/s.

Sending pre‑converted Buffers roughly doubles throughput because the server skips per‑request string‑to‑Buffer conversion.

Reference

http://nodejs.cn/api/buffer.html

深入浅出 Node.js Buffer

Do you want a better understanding of Buffer in Node.js? Check this out.

A cartoon intro to ArrayBuffers and SharedArrayBuffers

buffer.js v10.x

本文已获作者 “五月君” 授权转载，原文发表于公众号 “Nodejs技术栈”，可以点击原文查看。

欢迎各位投稿。