Microsoft released MAI-Transcribe-2-Streaming for real-time transcription with 100ms latency, plus MAI-Voice-2.1 and Flash for synthesis; the API streams revisable intermediate results, requires apps to manage commit timing, supports 60 languages, and is in public preview on Azure Foundry.
This article explains a voice agent architecture that separates real-time full-duplex frontend interaction from asynchronous backend computation using delegation IDs, detailing the two-timeline mechanism, three-layer responsibility split, task state machine, and evaluation strategies to avoid stale responses and interruption mishandling.
Google's release of Gemini 3.8 Live and Extended Thinking models highlights a shift in voice agent competition from natural speech to reasoning, tool use, and visual perception, with major players pursuing divergent strategies in reasoning depth, ecosystem integration, consumer scenarios, and platform embedding amid rapidly growing market demand.
This guide walks through choosing the right voice‑agent architecture, implementing streaming ASR with WebSocket, triggering sentence‑by‑sentence TTS, wiring the three layers together via async generators, optimizing latency to under a second, and avoiding common pitfalls such as missing VAD and checkpoint persistence.
In a Silicon Valley tech conference, Mu Shen shared how voice agents—real‑time, task‑oriented AI—were applied to an open‑world game as an AI NPC and to a Fortune‑500 insurer as an AI tele‑salesperson, revealing technical challenges, model architectures, training strategies, evaluation methods, and key lessons for future deployments.
