How to Build a Fast Custom WebAssembly Frame Extraction with FFmpeg

Project Background

In video editors, extracting key frames after a user uploads a video is needed for quick cover selection. Traditional canvas‑based extraction suffers from limited format support, main‑thread DOM dependency, and imprecise seeking.

Problems of Traditional Approach

Limited to formats supported by the video tag (e.g., no FLV, MKV, AVI).

Runs on the main thread, affecting page performance when processing many frames.

Cannot precisely control frame selection; setting currentTime seeks to the nearest decoded frame rather than a true key frame.

Proposed Solution: FFmpeg + WebAssembly

Compile FFmpeg to WebAssembly and execute it in a Web Worker. The worker decodes video frames to rgb24, converts them to ImageData, and sends the results back to the main thread, achieving efficient and flexible frame extraction.

Understanding WebAssembly

WebAssembly (Wasm) is a binary instruction format for a stack‑based virtual machine, designed as a portable compilation target for programming languages.

Wasm provides a lightweight, portable VM similar to Docker containers, offering high performance, cross‑platform support, safety, and multi‑language portability.

How Chrome Executes Wasm

Chrome’s V8 engine uses tiered compilation: the baseline Liftoff compiler for fast initial compilation, and the optimizing TurboFan compiler for hot functions. Streaming compilation and code caching further reduce load time.

Frame Extraction Strategy

Goal: produce high‑quality cover images by extracting key frames. The requirement is 12 images per video. If fewer key frames exist, fill gaps by sampling every 2 seconds between key frames; if more, select evenly spaced frames. The first frame is always an I‑frame, so it is returned immediately to reduce perceived latency.

Custom FFmpeg Build

Use Emscripten to compile FFmpeg with only the needed libraries ( libavformat, libavcodec, libswscale, libavutil) to keep the Wasm payload small.

git clone https://github.com/emscripten-core/emsdk.git
cd emsdk
git pull
./emsdk install latest
./emsdk activate latest
source ./emsdk_env.sh

Configure FFmpeg:

CFLAGS="-s USE_PTHREADS"
LDFLAGS="$CFLAGS -s INITIAL_MEMORY=33554432"
CONFIG_ARGS=(
  --prefix=$WEB_CAPTURE_PATH/lib2/ffmpeg-emcc \
  --target-os=none \
  --arch=x86_32 \
  --enable-cross-compile \
  --disable-x86asm \
  --disable-inline-asm \
  --disable-stripping \
  --disable-programs \
  --disable-doc \
  --extra-cflags="$CFLAGS" \
  --extra-cxxflags="$CFLAGS" \
  --extra-ldflags="$LDFLAGS" \
  --nm="llvm-nm-12" \
  --ar=emar \
  --ranlib=emranlib \
  --cc=emcc \
  --cxx=em++ \
  --objcc=emcc \
  --dep-cc=emcc
)
emconfigure ./configure "${CONFIG_ARGS[@]}"

Compile to Wasm:

emcc $WEB_CAPTURE_PATH/src/capture.c $FFMPEG_PATH/lib/libavformat.a $FFMPEG_PATH/lib/libavcodec.a $FFMPEG_PATH/lib/libswscale.a $FFMPEG_PATH/lib/libavutil.a \
  -O0 -lworkerfs.js --pre-js $WEB_CAPTURE_PATH/dist/capture.worker.js \
  -I "$FFMPEG_PATH/include" -s WASM=1 -s TOTAL_MEMORY=$TOTAL_MEMORY \
  -s EXPORTED_RUNTIME_METHODS='["ccall","cwrap"]' \
  -s EXPORTED_FUNCTIONS='["_main","_free","_captureByMs","_captureByCount"]' \
  -s ASSERTIONS=0 -s ALLOW_MEMORY_GROWTH=1 \
  -o $WEB_CAPTURE_PATH/dist/capture.worker.js

Use workerfs in the Web Worker to handle large files without copying them into memory, preventing crashes on big videos.

JS ↔ C Communication

JavaScript calls C functions via Module.cwrap. C can invoke JavaScript using emscripten_run_script. Memory buffers are accessed through typed arrays such as Module.HEAPU32, with pointer arithmetic required for struct fields.

FFmpeg API Used

av_register_all

– register all codecs. avformat_open_input – open media file. avformat_find_stream_info – retrieve stream information. avcodec_find_decoder – locate decoder. av_read_frame – read compressed packets. av_seek_frame – seek to a timestamp (key frame). avcodec_send_packet / avcodec_receive_frame – decode frames.

Result and Benefits

Custom‑compiled FFmpeg Wasm reduces the bundle from ~24 MB (full @ffmpeg/ffmpeg) to ~4 MB while delivering faster key‑frame extraction and lower latency. The solution has been deployed in Baidu’s “BaiJiaHao” video platform for several months, showing clear performance improvements.

Project repository: https://github.com/wanwu/cheetah-capture