From Q&A to Real‑Time Seeing and Speaking: JD’s World‑First Open‑Source JoyAI‑VL‑Interaction

JD recently open‑sourced JoyAI‑VL‑Interaction, the world’s first full‑stack open‑source interaction model that moves large language models from a simple "question‑answer" paradigm to continuous "see‑and‑speak" behavior. The system allows developers to quickly build AI assistants that continuously observe video streams, make autonomous decisions about when to speak or stay silent, and hand off complex tasks to backend agents.

Why traditional multimodal models fall short

Most current multimodal models focus on parameter size, knowledge, and reasoning, but they still operate in a turn‑based Q&A mode: users upload an image or video, ask a question, and the model replies. This works for static analysis but fails in real‑world scenarios where timing is critical.

Three breakthroughs of JoyAI‑VL‑Interaction

Active judgment instead of passive answering – Unlike traditional models that wait for a user query, JoyAI‑VL‑Interaction continuously watches the video stream and decides autonomously when to speak or stay silent. For example, if a user sets a rule "alert me when the referee shows a red card," the model monitors the match and issues an instant warning without being asked.

Real‑time response rather than post‑event summarization – Conventional video understanding processes a completed video, which introduces latency. JoyAI‑VL‑Interaction processes live streams, enabling immediate reactions in security alerts, live translation, broadcast commentary, and operational guidance.

Intelligent delegation with ongoing observation – When the model encounters tasks such as code generation, tool invocation, or complex reasoning, it can delegate to a backend large model or agent while the front‑end continues to monitor the scene. The backend returns results that are seamlessly integrated into the ongoing dialogue, forming a "front‑end real‑time assistant + back‑end intelligent brain" collaboration.

Evaluation results

In a benchmark covering 58 real‑world blind‑review cases across scenarios like monitoring alerts, real‑time counting, translation, temporal awareness, and live tour commentary, JoyAI‑VL‑Interaction achieved a 77.6% overall win rate against the Doubao video‑call assistant and an 87.9% win rate against the Gemini video‑call assistant. In monitoring‑alert tasks, it reached a 100% win rate, demonstrating the advantage of intrinsic proactive interaction over external rule‑based triggers.

Open‑source assets

The project releases the complete technology stack, including model weights, interaction dataset, training recipes, and a deployable system. Developers can access the code, model, and dataset at the following links:

Code: https://github.com/jd-opensource/JoyAI-VL-Interaction

Model: https://huggingface.co/jdopensource/JoyAI-VL-Interaction-Preview

Dataset: https://huggingface.co/datasets/jdopensource/JoyAI-VL-Interaction

Supported inputs and extensibility

JoyAI‑VL‑Interaction supports various video sources (camera, live stream, surveillance feed), voice input/output, visual interfaces, long‑term memory, backend model APIs, and vLLM deployment. Components such as ASR, TTS, visualization, and external tools can be swapped out to fit specific services or front‑end applications.

Potential applications

The framework is not a closed product; it can be adapted for security monitoring, elderly or child care, live broadcast commentary, e‑commerce guidance, operational assistance, AI glasses, and accessibility aids for the visually impaired, among other real‑time AI assistant scenarios.

Broader context

This release follows JD’s recent open‑source milestones, including the JoyAI‑LLM Flash Instruct model and the JoyAI‑Image‑Edit model, and precedes the long‑video generation model JoyAI‑Echo. Together, these contributions position JD among the global leaders in model infrastructure and illustrate a strategic push to bring AI from the digital realm into the physical world.