How to Render YUV Video Frames to RGB with WebGL: A Step‑by‑Step Guide

Introduction

The previous article "Video Decoding" described how raw HEVC streams are decoded into YUV data; this article explains how to render the decoded YUV data into an image using WebGL.

What Is YUV?

YUV (also known as Y'UV, YCbCr, YPbPr) separates luminance (Y) from chrominance (U and V). Y renders grayscale images, while UV adds color. YUV formats come in two families:

Packed formats store Y, U, V together in macro‑pixel arrays, similar to RGB storage.

Planar formats store each component in separate planes; for example, I420 (4:2:0), YV12, IYUV.

The example video uses 420p sampling, so all subsequent code assumes YUV‑420p.

Difference Between YUV and RGB

RGB is an additive color model based on human perception of red, green, and blue light. YUV is designed for efficient compression: the human eye is more sensitive to luminance than chrominance, so YUV samples luminance fully and chrominance at lower rates (e.g., 4:2:0, 4:2:2, 4:1:1, 4:4:4).

Because YUV consumes less bandwidth while displays use RGB, video streams are typically transmitted in YUV and converted to RGB for rendering.

WebGL‑YUV Rendering

High‑performance YUV rendering on the web relies on WebGL, moving YUV‑to‑RGB conversion into a shader for GPU acceleration.

WebGL Basics

WebGL, derived from OpenGL, provides a gl context from an HTML5 canvas. Geometry is drawn using points, lines, or triangles; rectangles are composed of two triangles. Shaders—vertex and fragment—are written in GLSL and compiled into a program.

const gl = canvas.getContext('webgl');
// Create shaders
const vertexShader = gl.createShader(gl.VERTEX_SHADER);
const fragmentShader = gl.createShader(gl.FRAGMENT_SHADER);
const program = gl.createProgram();
if (!(vertexShader && fragmentShader && program)) {
    console.warn('shaders create failed');
}
// Compile vertex shader
gl.shaderSource(vertexShader, vertexShaderScript420);
gl.compileShader(vertexShader);
if (!gl.getShaderParameter(vertexShader, gl.COMPILE_STATUS)) {
    console.warn('Vertex shader failed to compile: ', gl.getShaderInfoLog(vertexShader));
}
// Compile fragment shader
gl.shaderSource(fragmentShader, fragmentShaderScript420);
gl.compileShader(fragmentShader);
if (!gl.getShaderParameter(fragmentShader, gl.COMPILE_STATUS)) {
    console.log('Fragment shader failed to compile: ', gl.getShaderInfoLog(fragmentShader));
}
// Link program
gl.attachShader(program, vertexShader);
gl.attachShader(program, fragmentShader);
gl.linkProgram(program);
if (!gl.getProgramParameter(program, gl.LINK_STATUS)) {
    console.log('Program failed to compile: ', gl.getProgramInfoLog(program));
}
gl.useProgram(program);

GLSL Concepts

Attributes and Buffers : Attributes describe how to fetch per‑vertex data from buffers (e.g., positions, texture coordinates).

Uniforms : Global variables set once per draw call, accessible in both vertex and fragment shaders.

Textures : Image data sampled in shaders; Y, U, and V planes are uploaded as separate textures.

Varyings : Variables passed from the vertex shader to the fragment shader.

Shader Code

Vertex shader (handles vertex positions and texture coordinates):

attribute vec4 vertexPos;   // Vertex coordinates
attribute vec2 texturePos; // Texture coordinates
varying vec2 textureCoord; // Pass to fragment shader

void main() {
    gl_Position = vertexPos;
    textureCoord = texturePos;
}

Fragment shader (converts YUV to RGB using a matrix):

// Set precision
precision highp float;
varying highp vec2 textureCoord;
uniform sampler2D ySampler;
uniform sampler2D uSampler;
uniform sampler2D vSampler;
const mat4 YUV2RGB = mat4(
    1.1643828125, 0, 1.59602734375, -.87078515625,
    1.1643828125, -.39176171875, -.81296875, .52959375,
    1.1643828125, 2.017234375, 0, -1.081390625,
    0, 0, 0, 1
);

void main(void) {
    highp float y = texture2D(ySampler, textureCoord).r;
    highp float u = texture2D(uSampler, textureCoord).r;
    highp float v = texture2D(vSampler, textureCoord).r;
    gl_FragColor = vec4(y, u, v, 1) * YUV2RGB;
}

Texture Mapping

After creating shader programs, vertex data and texture coordinates must be supplied. Vertex positions range from -1 to 1; texture coordinates range from 0 to 1. Because the canvas Y axis points downwards, either gl.pixelStorei(gl.UNPACK_FLIP_Y_WEBGL, 1) or a manual flip of vertex coordinates is required.

Three mapping cases were tested; the third case (flipping both vertex and texture coordinates) renders correctly without extra API calls.

// Create vertex buffer
gl.bindBuffer(gl.ARRAY_BUFFER, gl.createBuffer());
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
    1, 1, -1, 1, 1, -1,
    1, -1, -1, 1, -1, -1
]), gl.STATIC_DRAW);
const vertexPos = gl.getAttribLocation(program, 'vertexPos');
gl.enableVertexAttribArray(vertexPos);
gl.vertexAttribPointer(vertexPos, 2, gl.FLOAT, false, 0, 0);

// Create texture coordinate buffer
gl.bindBuffer(gl.ARRAY_BUFFER, gl.createBuffer());
gl.bufferData(gl.ARRAY_BUFFER, new Float32Array([
    1, 1, 0, 1, 1, 0,
    1, 0, 0, 1, 0, 0
]), gl.STATIC_DRAW);
const texturePos = gl.getAttribLocation(program, 'texturePos');
gl.enableVertexAttribArray(texturePos);
gl.vertexAttribPointer(texturePos, 2, gl.FLOAT, false, 0, 0);

Creating Textures

function createTexture(gl) {
    const texture = gl.createTexture();
    gl.bindTexture(gl.TEXTURE_2D, texture);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MAG_FILTER, gl.LINEAR);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_MIN_FILTER, gl.LINEAR);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_S, gl.CLAMP_TO_EDGE);
    gl.texParameteri(gl.TEXTURE_2D, gl.TEXTURE_WRAP_T, gl.CLAMP_TO_EDGE);
    gl.bindTexture(gl.TEXTURE_2D, null);
    return texture;
}
const yTexture = createTexture(gl);
const ySampler = gl.getUniformLocation(program, 'ySampler');
gl.uniform1i(ySampler, 0);
const uTexture = createTexture(gl);
const uSampler = gl.getUniformLocation(program, 'uSampler');
gl.uniform1i(uSampler, 1);
const vTexture = createTexture(gl);
const vSampler = gl.getUniformLocation(program, 'vSampler');
gl.uniform1i(vSampler, 2);

Drawing YUV Data

Using the YUV‑420p frame buffer obtained from FFmpeg, upload Y, U, and V planes to their respective textures and draw the two triangles that form a rectangle.

// Assume buffer contains decoded frame, videoWidth/videoHeight are dimensions
const size = videoWidth * videoHeight;
gl.viewport(0, 0, videoWidth, videoHeight);

// Y plane
const yLen = size;
const yData = buffer.subarray(0, yLen);
gl.activeTexture(gl.TEXTURE0);
gl.bindTexture(gl.TEXTURE_2D, yTexture);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.LUMINANCE, videoWidth, videoHeight, 0, gl.LUMINANCE, gl.UNSIGNED_BYTE, yData);

// U plane (1/4 size)
const uLen = size / 4;
const uData = buffer.subarray(yLen, yLen + uLen);
gl.activeTexture(gl.TEXTURE1);
gl.bindTexture(gl.TEXTURE_2D, uTexture);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.LUMINANCE, videoWidth / 2, videoHeight / 2, 0, gl.LUMINANCE, gl.UNSIGNED_BYTE, uData);

// V plane (same size as U)
const vLen = uLen;
const vData = buffer.subarray(yLen + uLen, yLen + uLen + vLen);
gl.activeTexture(gl.TEXTURE2);
gl.bindTexture(gl.TEXTURE_2D, vTexture);
gl.texImage2D(gl.TEXTURE_2D, 0, gl.LUMINANCE, videoWidth / 2, videoHeight / 2, 0, gl.LUMINANCE, gl.UNSIGNED_BYTE, vData);

// Draw two triangles (6 vertices)
gl.drawArrays(gl.TRIANGLES, 0, 6);

Conclusion

By following the steps above, YUV frames decoded with FFmpeg can be rendered onto a canvas using WebGL. This article summarizes the author's experience implementing a H.265 player on the web; feedback and corrections are welcome.

Stay tuned for the next article in the series: "From Zero to One: Web H.265 Player – MP4/fMP4 Demuxing".