How to Turn FunASR into a Production‑Ready Real‑Time Speech Platform: From Single‑Node Demo to Million‑Scale Architecture

Introduction

When a demo works, the problem solved is "model usability"; when a system goes live, the challenge becomes "business deliverability". The core question is how to evolve FunASR from a single inference program into a production‑grade real‑time speech platform that can handle millions of concurrent sessions.

Why FunASR Demo Fails in Production

Typical onboarding steps—download model, run demo, test locally—ignore production concerns such as queueing, GPU memory spikes, tail‑latency bursts, and out‑of‑order sessions. Production streaming ASR must meet low latency, high concurrency, high availability, high accuracy, low cost, and strong governance.

Core Principles of Streaming ASR

Chunk processing: audio is split into small time slices (e.g., 60‑200 ms) and processed incrementally.

Cache management: encoder, decoder, VAD, and partial‑text caches must be kept per session.

Session stickiness: all chunks of a session must be routed to the same state‑machine instance.

VAD stability: voice activity detection directly impacts perceived latency.

Architecture Evolution Stages

The system evolves through five stages:

Stage 0 – Single‑process prototype : model verification, client‑server testing, hot‑word design.

Stage 1 – Single‑machine multi‑process + Nginx : Nginx handles WebSocket upgrade, supervisor manages processes, local model cache reduces load.

Stage 2 – Separate ingress and inference layers : a gateway performs authentication, rate‑limiting, and routing to a session router, while ASR pods focus solely on inference.

Stage 3 – Event‑driven with Kafka and Redis : Kafka buffers audio chunks, provides ordering and back‑pressure; Redis stores session metadata, hot‑word configs, and distributed locks.

Stage 4 – Kubernetes with HPA/KEDA : GPU resources are pooled, horizontal pod autoscaling uses custom metrics (active streams, P95 latency), and KEDA scales offline workers based on Kafka lag.

Session Ordering and Governance

Streaming ASR is stateful; each session holds audio offset, caches, VAD state, partial/final text, hot‑words, and sequence numbers. Out‑of‑order routing breaks cache reuse, causing latency spikes and result mismatches. Three common implementations for ordering are:

Gateway local consistent hashing (fast but suffers hash churn on scaling).

Kafka partitioning by session_id (guarantees order and enables replay).

Dedicated session router service (clear logic, supports gray releases, but adds complexity).

The recommended stable combo is gateway routing + Kafka partitioning + Redis for metadata.

Production‑Level Code Walkthrough

Key components include:

Directory layout for gateway, router, workers, infra, and tests.

Gateway layer that authenticates, generates session_id, enforces per‑tenant quotas, and pushes audio chunks to Kafka.

Model pool with micro‑batching (configurable max batch size and wait time) to improve GPU throughput.

Session repository that creates or retrieves SessionState objects, validates sequence numbers, and tracks stage transitions.

Streaming worker that locks per session, builds inference requests with cached states, updates caches, measures latency, and emits partial or final results.

Model manager that performs warm‑up, limits inflight requests with a semaphore, and provides a simple batch generation stub.

Result format with fields session_id, sequence, result_type, text, confidence, is_final, version, and trace_id to guarantee idempotent updates.

Offline post‑processor that restores punctuation, normalizes numbers (ITN), and applies hot‑word corrections.

Scaling and Engineering Capabilities

Four‑level flow control is essential for high concurrency:

Tenant‑level connection quota.

Gateway‑level rate limiting per chunk.

Queue‑level back‑pressure (Kafka lag monitoring).

GPU inflight gate (semaphore).

Batch size should be small (4‑16) with a short wait window (5‑15 ms) to keep first‑token latency low. Model and hot‑word caches use a two‑level strategy: in‑process hot‑path cache plus Redis metadata for versioning.

GPU memory management includes warm‑up, inflight gating, exposing active‑stream metrics to HPA, and keeping a safety margin (~20 %).

Kubernetes Production Deployment

A sample Deployment defines 4 replicas, GPU node selector, environment variables for Redis, Kafka, and batch parameters, readiness/liveness probes that also check model warm‑up, and a preStop hook to allow graceful draining. A Service exposes port 80.

HorizontalPodAutoscaler uses custom metrics funasr_active_streams (target 60) and funasr_p95_latency_ms (target 800 ms) instead of CPU alone.

KEDA scales offline workers based on Kafka lag (e.g., lag > 200 triggers scaling up to 50 replicas).

Monitoring and Observability

Core business metrics to expose via Prometheus include:

active_sessions

first_token_latency_ms

final_latency_ms

partial_update_interval_ms

rtf (real‑time factor)

queue_wait_ms

gpu_inflight

finalization_rate

hotword_hit_ratio

Technical metrics such as GPU memory/SM utilization, pod restarts, Kafka lag, Redis RTT, disconnect rate, and session‑reclaim time are also required. Logs must carry trace_id, session_id, tenant_id, sequence, chunk_size, result_version, worker_id, and model_version for end‑to‑end tracing.

Real‑World Business Case: Call‑Center Quality Inspection

Initial architecture (client → gateway → direct ASR) suffered OOM, session mis‑ordering, unreclaimed sessions, and hot‑word reload latency. After refactoring to the multi‑layer design with Kafka, Redis, per‑tenant quotas, and enriched monitoring, the system achieved:

P95 first‑token latency ↓ from 1.8 s to 420 ms.

P99 final latency ↓ from 4.7 s to 1.3 s.

GPU utilization ↑ from 28 % to 63 %.

OOM incidents eliminated.

Hot‑word recall ↑ ~11 %.

The platform now operates under metric‑driven governance rather than manual firefighting.

Designing for Million‑Scale Concurrency

Four capacity layers must be sized independently:

Connection layer (WebSocket count, heartbeat, TLS cost).

Message layer (Kafka partitions, key distribution, lag).

Inference layer (GPU inflight capacity, batch efficiency).

Governance layer (tenant quotas, hot‑word propagation, gray releases).

The key principle is to separate and independently scale each layer instead of merely adding more GPUs.

Load‑Testing Methodology

Load tests must simulate full session behavior: connection establishment, chunk streaming at varying speech rates, silence patterns, network jitter, and reconnection. Recommended metrics include max stable active streams per instance, P50/P95/P99 first‑token latency, final latency, queue wait time, GPU utilization, disconnect rate, and session leak rate. Test scenarios cover steady‑state, spike, long‑duration, and failure injection.

A sample k6 script creates WebSocket sessions, sends binary chunks, and ends with a JSON {"type":"end"} message.

Production Checklist

Model warm‑up completed and inflight limit configured.

Separate online and offline models, hot‑word cache versioned.

Globally unique session_id, per‑chunk sequence, result versioning, and explicit session finalization.

Session ordering guaranteed, back‑pressure buffers in place, per‑tenant rate limiting, auto‑scaling enabled.

Monitoring of latency, queue, inflight, GPU, and per‑tenant metrics; traceability for each session.

Graceful pod termination, Kafka lag‑driven degradation, Redis fallback, rapid OOM instance removal.

Common Pitfalls and Anti‑Patterns

Treating streaming ASR as a simple HTTP endpoint (ignores state and ordering).

Storing high‑frequency tensor caches in shared Redis (causes latency spikes).

Maximizing single‑instance throughput at the expense of tail latency and reliability.

Focusing on average latency instead of P95/P99 and first‑token latency.

Mixing online real‑time and offline post‑processing pipelines.

Conclusion

Production‑grade FunASR is not about running a model; it is about building a system where massive real‑time sessions run reliably, ordered, and with low latency under controlled cost. By addressing session state, multi‑layer scaling, observability, and governance, a demo can be transformed into a robust platform that powers use cases such as live subtitles, call‑center QA, and real‑time translation.