Why Real-Time Agents Need More Than One Loop: Google’s AMIE Splits Talk, Think, and See

Real‑time agents encounter a growing system dilemma: interaction must be fast, reasoning can be slow, and perception must run continuously. A traditional single‑agent, single‑context, single‑loop design quickly becomes a bottleneck when low‑latency dialogue, deep clinical reasoning, and ongoing audio‑video analysis are required together.

1. AMIE (Video) solves more than just “making AI see video”

Google’s AMIE (Video) answers this by abandoning a monolithic agent and introducing three asynchronous agents—Talker, Planner, and Perception—each handling a distinct computational rhythm. Talker provides the fast path for user interaction, Planner performs slow, background reasoning, and Perception continuously processes audio‑video signals.

2. Why a single loop cannot sustain the workload

In a conventional agent, input enters a Context, the model reasons, produces a response or tool call, and then waits for the next input. This works for coding, research, or data‑analysis agents where users tolerate seconds‑long pauses. Real‑time agents, however, lose naturalness if they pause after each user utterance, and they cannot allocate enough compute to deep diagnostic reasoning while also monitoring ongoing visual cues.

3. Asynchronous decomposition of the three goals

Talker runs on a small compute budget, processing the most recent ~5 seconds of video and using the latest state from Planner and Perception. Planner maintains a symptom summary, differential diagnosis, and care plan, updating its state at most once every 10 seconds to balance load. Perception runs Gemini 3.1 Pro on longer audio‑video windows, storing persistent observations (e.g., a cough at the start of a session) for later reasoning.

4. Component ablation and performance impact

In 20 multi‑turn simulated scenarios, the full AMIE system achieved an overall case‑specific rubric score of 87 % (Talker‑only 71 %). Automatic single‑round accuracy was 59 % for the full system versus 42 % for Talker‑only; Perception contributed most to visual‑inspection tasks, while Planner boosted clinical reasoning and dialogue completeness.

5. Latency reduction through async orchestration

Switching from sequential to asynchronous orchestration reduced average single‑round latency from 21.4 seconds to 2.6 seconds. The improvement stems from removing blocking dependencies: Talker can answer immediately while Planner and Perception continue processing in the background.

6. Generalizing the time‑scale split

The same conflict appears in voice assistants (immediate reply vs. background retrieval), meeting agents (real‑time transcription vs. agenda summarization), robots (sensor‑driven reactions vs. long‑term planning), and customer‑service agents (continuous intent monitoring vs. database queries). All these tasks operate on different time scales and benefit from separating fast‑path, slow‑reasoning, and continuous‑perception components.

7. Implications for future agent scaling

Agent scaling is shifting from merely increasing model size or context length to designing runtimes that schedule heterogeneous compute. Front‑end modules use small budgets for rapid response, while back‑end modules allocate larger budgets for deep reasoning, and perception modules maintain a persistent world model. This architectural shift is essential for maintaining performance as test‑time compute moves out of the user‑visible critical path.

8. Limitations

AMIE (Video) remains an experimental system evaluated on simulated clinical scenarios with professional actors, not real patients. It still struggles with fine‑grained visual localization, subtle emotion detection, and high‑frequency motion perception, and the underlying Project Astra prototype can affect dialogue naturalness. Therefore, the work does not claim that AI can replace doctors, but it highlights the need for multi‑agent, time‑scale‑aware runtimes in real‑time AI applications.