Why AI Assistants Shouldn't Just Wait for Questions: Insights from Tsinghua’s EgoIntrospect and IPIBench

Background: From Question‑Answering to Collaborative Assistants

Recent work by Thinking Machine Lab highlighted that most AI systems still operate in a turn‑based Q&A mode, which does not reflect real human collaboration that involves pauses, interruptions, and timing adjustments. To move AI assistants toward genuine collaboration, the Tsinghua MEOW Lab together with partners introduced two studies addressing user understanding and proactive interaction.

EgoIntrospect: Enabling AI to Truly Understand Users

Traditional multimodal models can recognize objects, actions, and scenes, but a wearable assistant must also infer the user's internal state. EgoIntrospect collects egocentric data from 60 participants over more than 180 hours using smart glasses, watches, wristbands, and rings, capturing video, audio, eye‑tracking, and physiological signals. Participants self‑annotate important moments and later add labels for emotions, intentions, and memory needs.

Based on this dataset, three benchmark tasks are defined:

Emotion experience: decide which segments merit recording and predict the user's likely emotion and intensity.

Interaction intent: (a) in passive response, determine which external tools are needed to fulfill a request; (b) in proactive mode, identify meaningful interactions, assess helpfulness, and choose a non‑disruptive timing.

Cognitive memory: distinguish information the user can retain versus information that requires AI assistance, and specify appropriate retention duration.

This shifts evaluation from merely “seeing the scene” to interpreting the significance of visual and sensory cues for the user.

IPIBench: Proactive Interaction Under Continuous Video Streams

While EgoIntrospect focuses on user understanding, IPIBench evaluates when an AI should speak up. The benchmark simulates real‑time video streams where user commands can change at any moment. Models receive only past video frames and must perform three intertwined tasks: proactive monitoring, proactive task management, and instant Q&A.

For example, in a kitchen scenario a user might say, “Remind me when the water boils.” The model must wait for the water to boil before reminding, update or cancel the reminder if the user changes their mind, and avoid premature or delayed prompts.

Evaluation results show current multimodal large models struggle with stable proactive triggering and multi‑turn coordination.

IPI‑Agent: A Light‑Weight Interaction Scheduler

To address these shortcomings, the authors propose IPI‑Agent, which adds an external interaction‑scheduling layer without retraining the base model. It separates user input into queries, new tasks, and task modifications/cancellations, maintains an explicit task memory, and applies a time‑gate that first generates candidate responses from historical tasks and recent video, then decides whether the timing is appropriate for activation.

Conclusion and Outlook

Combined, EgoIntrospect and IPIBench move AI assistant evaluation from static video‑question answering toward continuous, context‑aware collaboration. In wearable glasses, smart watches, and embodied robots, assistants must not only generate fluent answers but also deeply understand user emotions, manage tasks reliably, and intervene at the right moment.