Audio Architecture and Quality Optimization in WebRTC: Devices, 3A Processing, Codec, NetEQ and Scenario‑Based Solutions

Background

WebRTC is the most popular open‑source framework for real‑time audio and video. After Google open‑sourced the GIPS engine in 2010, it became a W3C standard and, since January 2021, an official standard of both W3C and IETF. Although free and open‑source, WebRTC is complex, has a steep learning curve, and lacks built‑in server solutions, leaving room for commercial RTC PaaS providers.

1. WebRTC Audio Architecture

The audio pipeline consists of an upstream link and a downstream link :

Upstream: device capture → software 3A (AEC, ANS, AGC) → audio encoder → RTP packetization → SFU. Capture runs in a push mode (10 ms callback) and is usually handled on the capture thread.

Downstream: RTP reception → NetEQ (jitter buffer, sorting) → audio decoder → post‑processing (PLC, acceleration) → mixer → render thread. Render runs in a pull mode (10 ms pull).

2. Factors Influencing RTC Audio Quality

The full‑link quality depends on four main components: audio devices, 3A processing, audio codec, and NetEQ.

2.1 Audio Devices

Device characteristics differ across platforms, requiring tailored configurations.

2.1.1 Android

Audio drivers: Java AudioRecord/AudioTrack, C++ OpenSL ES, and the newer AAudio API (high‑performance, low‑latency).

Typical parameter configuration (JSON example):

{"audioMode":3,"audioSampleRate":48000,"audioSource":7,"query":"oneplus/default ","useHardwareAEC":true,"useJavaAudioClass":true}

OpenSL ES is widely used because the AudioDeviceModule (ADM) in WebRTC is a C++ layer that reduces Java‑JNI overhead. Stereo capture provides richer spectra than mono.

Hardware 3A settings (example):

Hardware 3A: audioMode = MODE_IN_COMMUNICATION, audioSource = VOICE_COMMUNICATION, streamType = STREAM_VOICE_CALL

Software 3A: audioMode = MODE_NORMAL, audioSource = MIC, streamType = STREAM_MUSIC

2.1.2 iOS

iOS uses AudioUnit. The common configuration is:

Hardware 3A: kAudioUnitSubType_VoiceProcessingIO + AVAudioSessionModeVoiceChat

Software 3A: kAudioUnitSubType_RemoteIO + AVAudioSessionModeDefault

iOS provides three I/O units: Remote I/O (default), Voice‑Processing I/O (adds AEC, AGC, ANS), and Generic Output (offline processing).

2.1.3 Windows

Typical drivers: DirectSound (DSound), CoreAudio, and WAV. Many laptops have built‑in microphone arrays with hardware‑based audio enhancement (often limited to 8 kHz).

2.1.4 macOS

Less common; supports similar analog gain and volume controls as Windows.

2.2 3A Processing (AEC, ANS, AGC)

3A is the front‑end pre‑processing chain that removes echo, suppresses noise, and balances gain.

Acoustic Echo Cancellation (AEC)

Uses the far‑end reference signal to model the echo path and subtract the estimated echo from the microphone signal. Challenges include hardware AEC variability, double‑talk, delay estimation, and residual echo.

Automatic Noise Suppression (ANS)

Based on Wiener filtering; estimates noise per frame and applies a multi‑feature probability model. Music is more vulnerable to ANS because background music often has low SNR in high frequencies.

Automatic Gain Control (AGC)

Adjusts gain after AEC/ANS, typically with VAD to avoid amplifying noise. Key parameters:

targetLevelDbfs – desired output level (e.g., –1 dB)

compressionGaindB – maximum gain (e.g., 12 dB)

AGC modes (enum):

enum { kAgcModeUnchanged, kAgcModeAdaptiveAnalog, kAgcModeAdaptiveDigital, kAgcModeFixedDigital };

PC platforms usually use kAgcModeAdaptiveAnalog (combined analog and digital gain).

2.3 Codec – Opus

Opus (USAC) combines SILK (speech) and CELT (music). It supports:

Bitrates 6 kbps–510 kbps

Sample rates 8 kHz–48 kHz

Frame sizes 2.5 ms–60 ms

CBR/VBR, mono/stereo, up to 255 channels

Robust PLC and packet loss concealment

Higher complexity CELT with higher bitrate yields better music quality; low‑complexity SILK is sufficient for voice‑only scenarios.

2.4 NetEQ

NetEQ handles jitter buffering, packet loss concealment (PLC), and time‑scale modification (acceleration/deceleration). It consists of an MCU (buffer management) and a DSP (decoding, PLC, etc.). When true packet loss occurs, NetEQ applies PLC to synthesize missing audio, but traditional PLC is limited to ~40 ms loss.

Improvement directions:

Use RED/FEC + NACK for weak‑network resilience.

Optimize PLC algorithms for longer loss periods.

3. Business‑Scenario‑Based Audio Solutions

3.1 Live Streaming

High‑fidelity music requires software 3A, a media volume bar, and music‑optimized codec settings. When an external sound card is attached, 3A is often bypassed.

3.2 Meeting

Typically uses hardware 3A, a call‑volume bar, and voice‑oriented codec. Some apps provide a “high‑fidelity music mode” that disables ANS.

3.3 Communication (e.g., WeChat video call)

Same as meeting: hardware 3A + call‑volume bar.

3.4 Online Education

General education: hardware 3A for clear speech.

Music education: software 3A + music‑focused codec (similar to live streaming).

3.5 "Watch‑Together"

Two SDKs (player + RTC) cause echo of the video audio. Three solution paths:

Hardware 3A only – simple but suffers from dual volume bars and inconsistent AEC across devices.

Software AEC with reference from the player – unifies volume control and improves echo removal.

Advanced version of #2 with synchronized playback via RTC signaling for optimal performance.

3.6 Case Study

A live‑streaming host using an external sound card experienced poor music quality. Root causes:

Software ANS degrading music.

Server‑side AAC LC at 64 kbps (single‑channel) truncating the frequency band to 16 kHz.

Solution: disable ANS for music, switch to higher‑bitrate AAC HE or Opus with music‑optimized settings.

3.7 Industry Trend

Hardware 3A is low‑power and widely deployed, but its quality varies across devices. Mature software 3A solutions now adapt to content type (speech vs. music) and are becoming the primary direction, offering consistent high‑quality audio with lower integration cost.

4. Summary

The article dissected the WebRTC audio chain—from device capture, through 3A processing, codec selection, and NetEQ handling—highlighting how each stage influences perceived audio quality. Scenario‑specific strategies were presented for live, meeting, communication, education, and watch‑together use cases, followed by a trend analysis pointing toward pure‑software 3A as the future of RTC audio.