Designing a Live‑Streaming Platform for 1.2 Million Concurrent Viewers

Overall Architecture

Three core requirements drive the design: sub‑200 ms latency, support for 1 million concurrent viewers, and high availability. The solution adopts a three‑layer “push‑stream‑transcode‑distribution” architecture with an independent Go‑based WebSocket message subsystem, physically separating video traffic from chat and operational traffic.

Push Stream and Transcoding

Protocol Selection

RTMP is discarded because it lacks HEVC/AV1 support, performs poorly on UDP‑based weak networks, and is no longer maintained. The primary ingest protocol is SRT (Secure Reliable Transport) , supplemented by WHIP (WebRTC‑HTTP Ingestion Protocol) for browser‑based publishing. SRT’s UDP foundation with built‑in ARQ and FEC keeps latency under 200 ms even with 5 % packet loss; WHIP enables WebRTC‑based H5 publishing.

Transcoding Solution

Each 1080p source is transcoded into 1080p, 720p, 480p and 360p with ABR segmentation. NVIDIA T4 GPUs paired with FFmpeg 6.x encode to AV1 , which reduces bitrate by roughly 30 % compared with H.265 at equal visual quality. A single machine equipped with four T4 cards handles about 60 concurrent 1080p streams; scaling to the target of 1 million concurrent viewers and 1 000 simultaneous live rooms requires roughly 70 such transcoding nodes.

Tiered GPU allocation is essential: top‑tier streamers receive dedicated GPU resources, while lower‑tier rooms share GPUs. A previous mistake of pooling all rooms on a single GPU pool caused a high‑profile stream to exhaust the pool and spike latency for all other rooms.

CDN Distribution and Edge Computing

Distribution Strategy

Delivering ~3 Tbps (1 million × 3 Mbps AV1 streams) cannot rely on origin servers alone. The design combines a self‑built origin with a multi‑CDN hybrid scheduler. Three CDN providers are integrated; a custom scheduler performs real‑time quality probing and traffic switching, avoiding single‑provider lock‑in demonstrated by a 40‑minute outage of a major CDN in a previous project.

Playback Protocol

Playback uses LL‑HLS (Low‑Latency HLS) with CMAF segmentation of 200 ms per chunk. Compared with traditional HLS (6‑10 s latency), LL‑HLS achieves 1.5‑2 s end‑to‑end latency, sufficient for e‑commerce live sales. For ultra‑low‑latency interactive scenarios (e.g., PK), WebRTC P2P is employed.

Edge Computing

Edge nodes perform on‑demand transcoding from the origin AV1 stream to LL‑HLS or FLV, reducing back‑haul bandwidth. Lightweight image‑enhancement models (RAISR) run at the edge, making 360p streams visually comparable to 480p, improving experience for low‑end devices.

Million‑Scale WebSocket Message System

Connection Layer

A Go gateway optimized with epoll sustains 500 k long‑lived connections on a 64 GB machine. Deploying eight such gateways (each below 60 % load) provides redundancy and load balancing.

Message Broadcasting

Chat follows a “write‑few, read‑many” pattern. Broadcasting a single comment to a 500 k‑viewer room is achieved through hierarchical aggregation:

Comments are first published to a NATS JetStream queue.

A routing layer hashes by room ID to shard messages.

The gateway batches up to 500 messages per second before pushing.

Clients render at most three comments per frame.

Testing shows a 500 k‑viewer room with 2 000 comments per second achieves P99 end‑to‑end latency under 800 ms.

Online‑viewer counting uses HyperLogLog for approximate counts with ~0.8 % error, avoiding the latency of Redis SCARD on million‑size sets.

Elastic Scaling and Disaster Recovery

Scaling

Traffic exhibits a tidal pattern (peak 20:00‑23:00, near‑zero at night). Kubernetes with a custom HPA drives autoscaling: GPU utilization triggers transcoding node scaling, while connection count drives gateway scaling. Scale‑out is configured for a 30 s response; scale‑in uses a 10 min cooldown to prevent oscillation.

Pre‑warming is critical: five minutes before a major streamer goes live, operators trigger a pre‑warm action that spins up transcoding instances and CDN edge nodes, cutting first‑frame load time by 60 %.

Disaster Recovery

A fiber‑cut incident caused a 12‑minute outage. The response was a dual‑datacenter active‑active deployment with BGP Anycast routing; DNS failover completes within five seconds. TiDB provides cross‑datacenter Raft‑based consistency for the database layer, while NATS multi‑cluster interconnection replaces Kafka for simpler configuration.

Key Performance Indicators and Tuning Details

First‑frame time – Target < 800 ms, actual P50 420 ms. Technique: CDN pre‑warm + key‑frame cache.

End‑to‑end latency – Target < 3 s, actual P95 1.8 s. Technique: LL‑HLS + CMAF 200 ms segments.

Stall rate – Target < 1 %, actual 0.6 %. Technique: ABR + edge quality enhancement.

Message latency – Target < 1 s, actual P99 780 ms. Technique: NATS + batch push.

System availability – Target 99.95 %, actual 99.97 %. Technique: active‑active + multi‑CDN.

Additional tuning:

GOP cache : cache the most recent GOP (~2 s) at the origin; new viewers receive the cached GOP, halving first‑frame time.

Weak‑network adaptation : clients report network quality every 2 s; the server adjusts bitrate based on RTT and loss, using LL‑HLS Rendition Report for seamless switches.

Memory‑pool reuse : the gateway reuses connection objects and buffers via sync.Pool, reducing GC pause from 50 ms to 3 ms.

Conclusion

System difficulty lies in coordinating all components. Push‑stream selection determines transcoding input format; transcoding output influences CDN caching strategy; CDN coverage constrains playback protocol choice. Any weak link degrades overall experience.

Core takeaways derived from the implementation:

Replace RTMP with SRT for better weak‑network performance and sub‑200 ms latency.

Adopt AV1 to cut bandwidth by ~30 % while retaining hardware decode support.

Deploy the message system independently from video flow.

Multi‑CDN scheduling is essential; reliance on a single provider introduces outage risk.

Fast scale‑out and slow scale‑in are key to stable elastic capacity.

Full‑chain load testing uncovered OOM in the message gateway, CDN edge‑node back‑source storms, and other hidden issues before production.