Cross‑Platform WebRTC Video Rendering: OpenGL, Metal, Vulkan & Direct3D

Introduction

WebRTC (Web Real‑Time Communication) is an API that enables real‑time audio and video communication in web browsers. The standard was finalized on January 26, 2021, making it an official internet communication standard. This article focuses on the video rendering techniques used on various platforms.

Background

There is currently no comprehensive summary of WebRTC video rendering across platforms. The author, who works on video post‑processing and up‑scaling on mobile devices, uses this opportunity to consolidate the relevant knowledge.

Video Pipeline

Video rendering must support video capture, pre‑processing, post‑processing, and decoding to the output format.

Rendering Process

CPU computes the content to be displayed and transfers it to the GPU via the data bus.

GPU renders the data and stores it in the frame buffer.

The video controller reads the frame buffer line‑by‑line according to the VSync signal and sends the data to the display.

Platform Rendering Technologies

Apple

Apple supports OpenGL and Metal. Both iOS and macOS implement OpenGL and Metal rendering in WebRTC. Metal has evolved from WWDC 2014 to WWDC 2021, adding performance shaders, tessellation, and GPU‑accelerated compute.

Android

Android supports OpenGL ES and Vulkan (available since API 24). WebRTC currently implements OpenGL rendering on Android, with plans to add Vulkan support.

Windows

Windows supports OpenGL, Direct3D, and Vulkan. WebRTC uses the open‑source SDL framework for rendering, but a Direct3D implementation is under consideration.

OpenGL

OpenGL is a cross‑language, cross‑platform graphics API for 2D/3D rendering. OpenGL ES is a subset designed for mobile devices. On iOS, an EAGLContext manages the OpenGL ES rendering context, while on Android GLSurfaceView encapsulates EGL and provides a convenient rendering surface.

Metal

Metal is Apple’s low‑level graphics and compute API that reduces CPU overhead and provides efficient multithreaded command buffers, resource management, and unified graphics‑compute pipelines. It includes encoders such as MTLRenderCommandEncoder, MTLComputeCommandEncoder, and MTLBlitCommandEncoder.

Vulkan

Vulkan is a cross‑platform 2D/3D graphics API that offers low CPU overhead and high performance. It supports modern GPU features and provides a unified pipeline for graphics and compute workloads.

Direct3D

Direct3D is Microsoft’s low‑level graphics API. Versions 10 and 11 introduced advanced features such as tessellation, multithreaded rendering, and compute shaders, enabling high‑performance 3D graphics on Windows.

Conclusion

With the rise of AR/VR, efficient video rendering is increasingly important. Apple has shifted to Metal, Android is embracing Vulkan, and Microsoft continues to develop Direct3D. These three APIs represent the leading directions for high‑performance cross‑platform graphics rendering.