Decoding H.265 Live Streams in the Browser with WebAssembly and FFmpeg

Background

With the rapid evolution of live‑streaming technology, high quality, low bandwidth, and low cost have become key goals. While H.264 remains dominant, the newer HEVC (H.265) standard offers roughly double the compression efficiency, prompting Huajiao Live to explore its adoption for web playback.

HEVC Overview

HEVC (High Efficiency Video Coding) improves over H.264 by using flexible Coding Tree Units (CTU) instead of fixed macroblocks, offering more granular block partitioning. It also expands intra‑prediction modes from 9 to 35, adds eight inter‑prediction patterns, and introduces Sample Adaptive Offset (SAO) filtering, achieving 2%‑6% bitrate reduction at the cost of ~2% higher decoding complexity.

Hardware vs. Software Decoding

Hardware decoding (GPU, DSP, ASIC) provides better performance but suffers from limited device support and patent fees. Modern CPUs are powerful enough that software decoding of H.265 is increasingly viable, especially on browsers that lack native H.265 support.

Web‑Side Decoding Challenges

All major browsers lack native H.265 playback, so decoding must be performed in software. JavaScript‑based solutions like libde265.js are too slow and cause audio‑video sync issues. WebAssembly (Wasm) offers a faster, low‑level execution environment suitable for heavy decoding tasks.

Chrome Native Audio/Video Pipeline

the video tag creates a DOM object and instantiates a WebMediaPlayer the player requests media data via a buffer

FFmpeg demuxes and decodes the streams

decoded frames are handed to renderer objects

the video element displays video and audio output

Since Chrome already uses FFmpeg internally, we can leverage FFmpeg to decode H.265 streams.

HTTP‑FLV Extension

HTTP‑FLV is a low‑latency live‑streaming protocol that packages audio/video into FLV and transports it over HTTP. Although upstream FFmpeg does not support H.265 in FLV, a third‑party extension (e.g., Kingsoft Video Cloud) adds this capability, enabling our solution.

WebAssembly Compilation

By compiling a trimmed FFmpeg build with Emscripten ( emconfigure ./configure) and selecting only the necessary demuxers and decoders, the resulting Wasm binary is reduced to ~1.2 MiB (15% smaller than the unoptimized build), improving load time.

Architecture & Workflow

The system consists of three Web Workers:

Downloader : uses the Streams API to fetch the live HTTP‑FLV stream, reads chunks via ReadableStreamDefaultReader, and posts them (as Transferable objects) to the Decoder.

Decoder : receives raw data, allocates memory in the Wasm heap with Module._malloc, copies data via Module.HEAPU8.set, and invokes FFmpeg to decode. Decoded video (YUV420P) and audio (PCM) are delivered through callback functions and posted back to the main thread.

Main Thread : receives decoded frames, pushes video frames into a VideoQueue and audio frames into an AudioQueue, then renders video using WebGL (converting YUV to RGBA) and plays audio via the Web Audio API.

A circular buffer manages the continuous stream, with head and tail pointers tracking read/write positions to keep memory usage bounded.

Custom FFmpeg I/O

FFmpeg’s avio_alloc_context creates a custom AVIOContext that reads from the in‑memory buffer. The context supplies read_packet, write_packet, and seek callbacks, allowing FFmpeg to treat the live stream as a regular input source.

YUV Rendering

The decoded frames are in YUV420P format, which cannot be drawn directly to a Canvas. WebGL shaders convert YUV to RGBA on the GPU, avoiding costly CPU‑side conversion and leveraging hardware acceleration.

Results

On a MacBook Pro (2.2 GHz i7, 16 GB RAM) using Chrome, the solution maintains memory usage between 270 MB and 320 MB and CPU load around 40%‑50% during prolonged playback.

References

FFmpeg official site: http://ffmpeg.org/

FFmpeg ticket on HTTP‑FLV support: http://trac.ffmpeg.org/ticket/6389

WebAssembly: https://webassembly.org/

V8 engine: https://v8.dev/

Emscripten: https://emscripten.org/