AURA: Real-Time Video Understanding Shifts from Post-Play Q&A to Continuous Interaction

Recent advances in video multimodal large models (VideoLLM) have pushed performance on tasks such as video description, video QA, and temporal grounding, but most methods still follow an offline paradigm: cache the whole video, then process it once. This limits applicability to real‑time assistants, live streaming, robotics, and surveillance, where low latency and continuous perception are required.

AURA (Always‑On Understanding and Real‑Time Assistance via Video Streams), a joint effort by CUHK MMLab and Huawei XiaoYi Model Application Lab, addresses these limitations by building a unified end‑to‑end visual interaction framework that remains online, continuously receives video streams, understands scene changes, and decides when to stay silent or respond.

The paper identifies two new challenges for streaming video understanding: (1) the video stream and dialogue history grow unbounded, demanding efficient context management; (2) the model must not only answer questions but also learn when to speak, when to stay silent, and when to wait for more information. Existing approaches either separate trigger and main models—causing inconsistency—or adopt a unified architecture that favors continuous description and struggles with open‑ended QA over long interactions.

AURA’s goals are two‑fold: enable a single model to process video frames sequentially while autonomously choosing silence or answer, and ensure the system can handle unbounded growth of video and text inputs without degrading performance during long‑running sessions.

To achieve this, AURA redesigns the interaction pipeline. Instead of splitting “whether to respond” and “how to respond” across two models, a single unified model performs observation, judgment, and answer generation within the same internal state, improving consistency for complex open‑ended interactions.

AURA defines three streaming QA categories: Real‑Time Q&A (immediate answer based on current frame), Proactive QA (delay response until sufficient evidence appears), and Multi‑Response QA (provide multiple answers as a scene evolves). These categories shape the data construction and capability modeling of the system.

For context management, AURA introduces an Interactive Video Stream Context Management mechanism that chops the video into short time blocks and aligns each block with user input, model answer, and possible silence state, forming a continuous dialogue. To bound context length, a “dual sliding‑window” strategy keeps a recent 30‑second video window with an additional 15‑second buffer and retains the latest 10 QA pairs, preserving the most valuable interaction history.

The Coarse‑to‑Fine Data Engine builds training data in five stages: video preprocessing (collecting diverse videos, resampling to 2 FPS, encoding to H.264), QA synthesis (generating timestamped QA pairs for each streaming QA type), QA refinement (augmenting difficulty and re‑phrasing), streaming structuring (organizing samples into time‑block dialogues with dual‑window truncation), and quality verification (filtering samples lacking visual evidence, temporal alignment, or consistency). This pipeline yields roughly 115 k streaming video QA samples, 59 k offline video QA samples, and about 174 k total samples (≈1.2 billion tokens), fine‑tuning only the LLM part of the Qwen3‑VL‑8B‑Instruct base model.

To balance silence and speech, AURA proposes a Silent‑Speech Balanced Loss . Because silent tokens dominate in streaming scenarios, ordinary cross‑entropy would bias the model toward perpetual silence. The loss supervises all silence messages and only the final non‑silent answer, while down‑weighting silence targets, ensuring a more balanced decision policy. Ablation experiments show that reverting to standard cross‑entropy drops the OmniMMI overall score from 25.4 % to 16.4 % and reduces proactive‑alert ability (PA) to 0 %.

For real‑time deployment, AURA integrates video, ASR, and TTS into a closed‑loop system. It reuses KV cache and employs a buffered floating‑window strategy instead of naïve FIFO, reducing prefix changes and minimizing repeated computation. On two 80 GB accelerator cards, the system processes video at 2 FPS with end‑to‑end latency of approximately 312 ms (ASR ≈ 84 ms, model TTFT ≈ 75 ms, first‑token decode ≈ 60 ms, TTS first‑chunk ≈ 93 ms).

Benchmark results demonstrate state‑of‑the‑art performance: StreamingBench 73.1 % total score, OVO‑Bench 65.3 % total score, and OmniMMI 25.4 % total score, surpassing many open‑source baselines and even closed‑source models such as GPT‑4o and Gemini‑1.5‑Pro on several metrics. The paper notes a modest degradation on traditional offline video understanding tasks, reflecting a trade‑off between offline accuracy and online interactivity.

In summary, AURA moves video LLMs from post‑play analysts to always‑on visual assistants that continuously observe, understand when to stay silent, and proactively respond at critical moments, offering a comprehensive system solution for real‑time visual intelligence.