How Pipeline Parallelism Cuts AI Voice Latency Below 700 ms

Latency Goal and Breakdown

For a natural‑conversation experience the total time from the user finishing speech to hearing the first AI syllable must stay under 700 ms (500 ms feels natural). The article defines the five stages and their typical latencies: turn detection 50‑300 ms, STT 50‑100 ms, LLM first token 100‑200 ms, TTS first byte 40‑80 ms, and network 20‑50 ms, giving a feasible total of 260‑555 ms.

Serial vs. Parallel Execution

Most developers mistakenly run STT → LLM → TTS sequentially, causing 2‑3 s latency because each stage waits for the previous one to finish. The article shows a timeline comparison where parallel execution overlaps STT with LLM and LLM with TTS, reducing latency to 400‑600 ms.

Pipeline Parallelism Strategies

Streaming STT : output partial transcripts while the user is still speaking (saves 100‑200 ms).

Speculative LLM : start inference as soon as a stable partial transcript appears (saves 100‑300 ms).

Streaming LLM→TTS : send the first sentence to TTS as soon as it is generated (saves 200‑400 ms).

Streaming TTS : play the first audio byte immediately (avoids waiting for the whole utterance).

Turn Detection – The Hidden Bottleneck

The article emphasizes that turn detection (“Did the user finish speaking?”) dominates latency. Solutions compared include a fixed 700 ms silence threshold (+700 ms), VAD (+300‑500 ms), STT endpointing (+100‑200 ms), and Deepgram Flux EOT (<300 ms), the latter saving 200‑600 ms.

Recommended Budget Allocation (WebRTC scenario)

Turn detection 50‑200 ms – Deepgram Flux or STT endpointing.

STT confirmation 50‑80 ms – Deepgram Nova‑3 / Flux.

LLM first token 100‑150 ms – GPT‑5 mini, Gemini 3.1 Flash, Claude Haiku.

TTS first byte 40‑75 ms – Cartesia Sonic Turbo (latency‑first) or ElevenLabs Flash (quality‑first).

Network 20‑50 ms – WebRTC with regional deployment.

Total 260‑555 ms, comfortably under the 700 ms target.

Component Selection per Layer

STT layer : choose streaming, low‑latency services; Deepgram Nova‑3 is the 2026 default, switch to Flux if turn detection is the bottleneck.

LLM layer : prefer fast, “good‑enough” models (GPT‑5 mini, Gemini 3.1 Flash, Claude Haiku). Keep system prompts short; each extra 100 tokens can add noticeable delay.

TTS layer : prioritize latency with Cartesia Sonic Turbo (≈40 ms TTFA) or quality with ElevenLabs v3 (≈75 ms). Cost‑sensitive deployments can swap Cartesia for ElevenLabs (+35 ms).

Orchestration layer : use LiveKit, Pipecat, or Retell AI to stitch the stages into a pipeline; LiveKit or Pipecat for in‑house teams, Retell/Vapi for rapid rollout.

Production Default Stack (2026)

Benchmark consensus (FutureAGI, Inworld, Coval) recommends: Deepgram Flux (turn + STT) + GPT‑5 mini (LLM) + Cartesia Sonic Turbo (TTS) + Retell AI / LiveKit (orchestration) → end‑to‑end 500‑700 ms.

Common Pitfalls

Double noise reduction (WebRTC NS + RNNoise + server‑side Krisp) over‑suppresses speech.

Choosing the strongest LLM (GPT‑4, Claude Opus) whose first token exceeds 500 ms.

Waiting for the full LLM response before starting TTS, causing 1‑2 s delay.

Relying only on average latency; P95 can be >1.5 s, so monitor P50/P95/P99 and TTFA.

Ignoring barge‑in; need sub‑60 ms TTS stop and sub‑40 ms LLM cancel.

Pipeline vs. End‑to‑End Speech‑to‑Speech

End‑to‑end S2S solutions (e.g., GPT‑Realtime‑2, Gemini 3.1 Flash Live) can reach 300‑500 ms but are black‑box, harder to monitor, and lock you to a vendor. The pipeline approach offers per‑stage observability, replaceability, and auditability, which most production teams still prefer in 2026.

Conclusion

Achieving sub‑700 ms latency is not about a single faster model but about a co‑designed, parallel pipeline. Pipeline parallelism is an architectural choice; Deepgram Flux solves turn detection, Cartesia Sonic Turbo solves TTS first‑byte latency, and fast LLMs with short prompts keep inference delay low.