Why Real‑Time Agents Need Multiple Loops: Google’s AMIE Splits Talk, Think, See

When an AI system must respond instantly, perform slow clinical reasoning, and continuously perceive video and audio, the traditional single‑agent, single‑context, single‑loop architecture reaches its limits. The article frames this as a three‑fold conflict: low latency, heavy reasoning, and ongoing perception.

AMIE (Video) Architecture

Google’s AMIE (Video) replaces the monolithic loop with three asynchronous agents:

Talker : the fast‑path front‑end that handles real‑time dialogue, listening for user pauses and generating responses using a small compute budget on the most recent ~5 seconds of video.

Planner : the slow‑reasoning back‑end that maintains symptom summaries, differential diagnoses, and care plans, updating its state at most once every ten seconds.

Perception : a continuous perception module that processes longer audio‑video windows with Gemini 3.1 Pro, storing observations (e.g., a cough at the start of a session) for later reasoning.

These agents are divided by computational characteristics rather than narrative roles, allowing each to run at its appropriate time scale.

Empirical Evaluation

Component ablation on 20 multi‑turn simulated clinical scenarios shows:

Full system case‑specific rubric score: 87 % (Talker‑only: 71 %).

Automatic single‑turn accuracy: 59 % (Talker‑only: 42 %).

Perception contributes most to visual‑inspection tasks; Planner boosts reasoning and dialogue completeness.

Switching from sequential to asynchronous orchestration cuts average single‑turn latency from 21.4 s to 2.6 s, proving that performance gains stem from decoupled scheduling rather than reduced model thinking.

Broader Implications

The paper argues that as agents move from one‑shot prompt completion to continuous, real‑time environments (voice assistants, meeting summarizers, robotics, customer‑service bots), the same latency‑reasoning‑perception tension appears. The solution is to treat the agent runtime as a scheduler that allocates different compute budgets across time scales, maintains shared state, and handles state expiration.

Thus, the next scaling law for agents may shift from larger models to smarter runtime architectures that separate fast‑path, slow‑path, and continuous perception workloads.