Realtime-Bench: Real-Time Interactive Tasks Defeat Frontier Models Like GPT-6

Introduction

Large models can understand complex web structures and game rules, but can they act within fleeting time windows — such as snapping up a hot ticket or rotating a Tetris piece before it lands? To evaluate real-time interactive capability, researchers from NVIDIA, UT Austin, Northwestern, OSU, UIUC, and Steadyworks created Realtime-Bench and tested 12 frontier models. Under real-time progression, the top model (GPT-6 Astra) reaches only 44% task success.

Three Core Real-Time Capabilities

RT-Perceive : Observe environment changes, judge whether current state is stale and if action opportunities still exist.

RT-Deliberate : Adjust reasoning depth based on remaining time — decide how long to think and whether continued thinking is worthwhile.

RT-Act : Seize the action moment — decide whether to wait or act immediately, and choose an action that remains valid at execution time.

Models must first detect an opportunity (e.g., ticket release), then control thinking time according to the purchase window, and finally execute before the window closes. A failure at any stage loses the chance.

Benchmark Design: Time-Dependent Tasks

Realtime-Bench contains 30 web tasks and 20 game tasks across eight scenario categories (travel, e-commerce, services, security, arcade, puzzle, strategy, simulation). Every task has two properties: the environment evolves autonomously over time, and the goal must be achieved within a time limit. Acting too early or too late leads to failure even if the action itself is correct. Examples include real-time auctions where the model must bid before the countdown ends while opponents raise bids, and Snake where the snake moves continuously and the model must turn in time to eat the apple.

Clock Protocols: How Environment Time Advances During Reasoning

Realtime-Bench evaluates models under three clock protocols to isolate the effect of reasoning time:

Wall Clock : Environment advances by the actual wall-clock time the model spends generating a response.

Paused Clock : Environment is frozen during reasoning; advances only after the model finishes. This serves as an idealized control to separate reasoning ability from time pressure.

Token Clock : Environment advances proportionally to the number of tokens generated. This removes API latency variance and probes the model's intrinsic time awareness.

Experimental Setup

Twelve frontier models were tested, grouped into three categories:

Standard async models : GPT-6 Astra, GPT-5.6 Sol, Claude Fable 5.1, Claude Opus 5, Gemini 3.8 Flash, DeepSeek V4.1 Flash, Kimi-K3.

Real-time streaming models (optimized for low latency): GPT-Realtime-2, Gemini 3.1 Flash Live, Gemini 3.8 Live.

Decision models (output only decisions, no free text): OpenAI Decisions, Jev 1.13.

Because real-time tasks allow no retries, the primary metric is Pass^3 — the probability that a model succeeds on all three repeated runs of the same task, measuring reliability.

Results: Real-Time Performance Remains Low

Under Wall Clock (real inference latency), the best model GPT-6 Astra achieves only 36% Pass^3 at high reasoning intensity. Most models exceed 1 second latency; Jev achieves <1s latency but only 8% Pass^3. Low latency alone does not guarantee reliable real-time performance — models must both respond quickly and possess sufficient reasoning ability.

Reasoning Intensity vs. Success: A Critical Trade-off

Paused Clock (no time pressure): Increasing reasoning intensity improves success for all six models tested. E.g., GPT-5.6 Sol rises from 42% to 58%; Claude Opus 5 from 62% to 70%.

Token Clock (time advances with tokens): Effects diverge. Claude Opus 5 drops from 44% to 30% with more reasoning, while Kimi-K3 rises from 24% to 36%. Wall Clock shows similar divergence: GPT-5.6 Sol falls from 16% to 8%, but Gemini 3.1 Flash Live rises from 6% to 8%.

This demonstrates a fundamental trade-off in real-time tasks: continued thinking may yield a better action but can also cause the model to miss the execution window.

Reducing Reasoning Latency Helps but Is Not Sufficient

Under Token Clock, the simulated tokens-per-second rate was increased from 32 to 64, 128, and finally infinite (zero reasoning time cost). Overall, relaxing time constraints improves success rates: Claude Opus 5 rises from 30% to 62%, Kimi-K3 from 22% to 60%. However, not all models improve monotonically, and many tasks remain unsolved even when reasoning consumes no environment time . Lowering latency helps but does not fully solve real-time interaction challenges.

Conclusion and Future Directions

Realtime-Bench tasks require models to recognize changing states, allocate thinking time based on remaining windows, and decide when to wait versus act. Future models need more flexible coordination of these components: compare alternatives when time permits, act quickly as windows close, and revise stale plans when new information arrives.

References

Paper: https://arxiv.org/abs/2610.22155 Project homepage: https://realtimeagents.org Hugging Face paper:

https://huggingface.co/papers/2610.22155