Beyond Voice Commands: How Sensaro OS Gives Smart Homes Intent Understanding in 48 Hours

Why Make the Environment Understand People

Modern smart homes add devices and features but not interaction simplicity. Users still memorize device names, control entry points, and commands. Automation relies on rigid "if A then B" rules. The team concluded the future interaction entry point should be human intent, and the future interface should be environment state.

Example: a family discusses weekend trips — "How about Summer Palace?" "Happy Valley?" "Shanghai Disneyland?" These are not commands. The system should recognize an ongoing travel discussion, continuously evaluate candidate intents, pre-fetch low-risk data (weather, routes, ticket availability, vehicle charge), and only when a clear decision emerges ("Let's go to Happy Valley, depart 8:30 AM") generate an executable plan: arrange vehicle charging and navigation, adjust lights and AC for departure, check door locks and gas, and plan environment restoration for return.

From Understanding a Sentence to Understanding a Scene

Sensaro OS converts user intent into a structured Sensaro State containing five fields:

Goal : e.g., enter focus state

Scope : study environment, phone, related devices

Actions : adjust lighting, AC, notifications

Boundaries : family calls and safety alerts must not be blocked

Lifecycle : 90 minutes or until user exits

Intent is not a one-sentence-to-one-action mapping; it represents the desired environment end-state. Understanding requires multimodal context: who speaks, where, current activity, scene, and whether the utterance is discussion, trial, or decision.

Xiaomi MiMo models handle perception, intent understanding, and orchestration: Xiaomi MiMo-v2.5-asr: speech recognition, speaker diarization Xiaomi MiMo-v2.5: visual, pose, scene, and emotion understanding Xiaomi MiMo-v2.5-pro: intent understanding, candidate state maintenance, Sensaro State generation, replanning on execution issues Xiaomi MiMo-v2.5-tts: voice output for reminders, confirmations, action explanations

In the travel discussion, the system continuously maintains candidate intents and confidence, pre-fetches low-risk data, and only upon clear intent generates a full Sensaro State, passes safety checks, and executes on real devices. Device execution results and user feedback then feed back to refine environment state.

Why MiMo Can't Act Immediately After Understanding

Family conversations contain exploration, retraction, jokes, and multiple candidates. Acting on "Happy Valley" immediately would turn proactive intelligence into a new nuisance. Design principle: discuss don't disturb, decide then act .

During discussion, system only updates candidate intents and confidence, pre-queries low-cost data. Only when a clear decision is detected with clear action time, participants, and authorization does it generate an executable environment state.

Four judgment layers between "hearing" and "acting":

Decision clarity : candidate intent stability, action time and participant clarity; if divergence persists, wait or ask.

Operation risk : reversibility, financial/privacy/external impact, device state readability.

Confirmation needed : low-risk reversible actions (lighting, AC) auto-execute; high-risk (door locks, payments, gas) require deterministic rules and explicit confirmation; model only proposes.

Information sufficiency : if identity, intent, or environment data insufficient, continue observing, ask, or enter conservative low-risk state.

Sensitive data boundaries: process locally where possible; as system nears real life, capability boundaries must be explicit, final control returned to user. Deterministic safety actions (e.g., gas leak → auto shut valve) must be pre-coded rules, not ad-hoc model decisions.

Real devices need structured constraints, state machines, fault tolerance, deterministic rules. For unstable model JSON output, team built multi-layer parsing: standard JSON first, then extract from code blocks/text, then fallback to preset. Xiaomi MiMo API includes timeout and degradation to prevent single-point failure.

Resulting division: model handles open-world understanding and proposes plans; system decides execution, how to execute, and boundaries. A clear, stable, verifiable engineering boundary sits between them.

Making the Demo Run in 48 Hours

Challenge 1: Digital Twin Construction

3D digital twin needed to visualize model judgments, environment state changes, device executions in real time — not just a pretty demo. Pure code 3D scene building (world coordinates, furniture placement, lighting, physics, dynamic effects like AC airflow) exceeded 48-hour capacity. Team used AI Agent and OpenSpec to generate scaffolding and repetitive code; engineers focused on coordinates, boundaries, interaction logic. Debugging centered on coordinate drift and wall-penetration; solved by converting world coordinates to visualization view, calibrating each object, adding boundary checks. Stack: React 18, React Three Fiber, Drei, Zustand. Scenes converged to study, bedroom, living room. Virtual house and real devices share the same Sensaro State — house is virtual, state is real.

Challenge 2: Hardware Integration Setbacks

Planned Bluetooth module unavailable; replacement delayed. Team decoupled hardware from upper system, modularized with high cohesion/low coupling, mocked hardware APIs. Day 2 morning: sourced module from colleague, assembled, reflashed SoC firmware. 1:00 PM: Bluetooth module connected, basic functions worked. First integration test: misconfigured parameters caused prototype car to drive off desk, smash on floor, destroying car and damaging module. Team braved heavy rain to Zhongguancun electronics market, found two replacement modules at a closing shop. Back at office: swapped modules, re-tuned parameters, re-tested — model commands finally drove car stably. Team celebrated.

Challenge 3: Communication Protocol Limitation

System spanned frontend, backend, phone app, hardware. Initially used WebSocket for low latency. Competition environment's HTTP/HTTPS didn't support WebSocket. Waiting for ops support would cost too much time. Judgment: demo load limited, real-time needs acceptable; priority is stable end-to-end run. Downgraded to HTTP polling. Added HTTP interfaces to WebSocket-based services, but kept compatibility: prefer WebSocket, auto-fallback to HTTP if unsupported. Preserves real-time capability, avoids single-protocol dependency. Changes touched multiple modules; AI coding accelerated code, but most time spent on design decisions, interface clarification, integration testing. For 48-hour hackathons, adapting to constraints matters more than sticking to initial design.

Post-Hackathon Technical Reflections

1. Complete Closed Loop Matters More Than Generating Device Commands

Getting MiMo to output plausible device commands is just one step. Real difficulty: stitching "perception → world model → intent judgment → planning → safety check → execution → state readback → user feedback" into a continuous loop. Without state readback and feedback ingestion, it's a one-off demo, not a sustainable environment.

2. Probabilistic Cognition and Deterministic Execution Must Be Separated

Models understand and reason, but real-device execution cannot rest on probabilistic judgments. Team layered MiMo, Sensaro State, and policy kernel: model understands and proposes candidates; system handles permissions, risk, safety verification. Practice reinforced need for a clear engineering boundary between model and physical devices.

3. Multimodal Relevance Trumps Quantity

Adding more modalities (speech, video, person state, device data) doesn't guarantee better judgment. Sensaro OS filters information relevant to current intent, combining confidence, time, space, risk level. Key insight: multimodal value lies in task-relevant information, not volume.

48 hours turned an initial concept into a runnable, verifiable prototype through repeated problem decomposition and adaptation. The exercise clarified that the hard part of "LLM + smart home" isn't just intent understanding, but completing the perception-judgment-planning-execution-feedback loop and establishing clear engineering boundaries between model capabilities and real-world determinism. Future work: improve device onboarding and execution loop stability/universality, add more real devices, enhance state readback and rollback, boost multi-person multi-intent judgment.