How MOSS Enables Real‑Time Long‑Video and Cocktail‑Party Audio Understanding in Complex Real‑World Contexts

Over the past year large‑scale models have rapidly expanded their perceptual abilities, moving from speech and image recognition to video analysis and multimodal integration. However, the number of modalities a model can ingest measures only perceptual breadth, not true understanding; real‑world meaning emerges from the relationships among signals over time.

To address this, the MOSS team (MOSS‑Intelligence and OpenMOSS) released a series of models that aim to embed multimodal inputs into a unified, continuously evolving context. The flagship releases include the video understanding model MOSS‑VL‑Realtime and the multi‑speaker transcription model MOSS‑Transcribe‑Diarize‑0.9B .

MOSS‑VL‑Realtime: From Watching Recorded Video to Watching Live Streams

Traditional video understanding systems operate offline: they ingest a finished video, then answer questions, unable to react to new frames. MOSS‑VL‑Realtime is built for continuous streams, allowing the model to answer at any moment, stay silent when information is insufficient, and correct earlier answers as the scene evolves.

The interaction paradigm is described in three points:

Any‑time answering : users may query the video at any playback point and receive an immediate response based on the observed frames.

Proactive silence : when no salient event is present, the model inserts a <|silence|> token and continues observing.

Watch‑and‑answer with correction : new frames are continuously fed into the context; if a sudden scene change occurs, the model can interrupt or revise its prior answer.

These capabilities stem from a unified token stream that interleaves video frames (anchored with absolute timestamps), user questions, and model replies, enabling a live‑chat‑like experience.

Architectural Innovations

Cross‑attention architecture : decouples visual encoding from language inference, reducing latency for streaming inputs.

Absolute timestamps : each sampled frame is represented by a dedicated token linked to its exact time.

XRoPE : maps text tokens and video patches into a shared three‑dimensional coordinate space.

256K context window : allows hour‑long video to fit into a single context.

Full‑stack engineering optimizations further improve throughput: FlashAttention now supports Cross‑Attention‑Mask, enabling training on sequences of up to 2048 frames and sampling rates up to 16 fps for fast‑changing scenes. Inference speed on an NVIDIA 4090 gains a 4.57× boost over vanilla Transformers and a 5.48× advantage over Qwen3‑VL when using the SGLang runtime.

MOSS‑Transcribe‑Diarize‑0.9B: End‑to‑End Multi‑Speaker Long‑Audio Transcription

Traditional pipelines chain separate ASR, speaker‑separation, and alignment modules, propagating errors across stages. The closed‑source MOSS‑Transcribe‑Diarize unified transcription, speaker labeling, and timestamp prediction into a single autoregressive generation task. The newly open‑sourced 0.9 B version combines a Whisper‑Medium encoder with a Qwen3‑0.6B‑style causal decoder, embedding text, speaker IDs, and timestamps as tokens.

Key capabilities include:

Processing up to ~90 minutes of audio in one pass (128 K context) without chunking.

Maintaining consistent speaker tags and timestamps across long dialogues.

Generating output at ~100 tokens/s (RTF 0.017) on a single 4090 GPU, transcribing 5–10 min audio in ~30 s.

On the AISHELL‑4 multi‑speaker benchmark, the model achieves a character error rate (CER) of 14.19 % and a multi‑speaker CER (cpCER) of 14.98 %, a Δcp of only 0.79, outperforming competing open‑source systems by more than 40 % relative reduction.

Hot‑word enhancement allows users to pre‑define names, product codes, or domain terms to improve specialized vocabulary accuracy.

Product Platforms and Ecosystem

Beyond model releases, MOSS‑Intelligence launched the Mossland AIGC audio‑video creation platform and the Moss Open Platform . Mossland aggregates the MOSS‑TTS family (over a thousand voice styles, zero‑shot voice cloning, real‑time streaming) with video and image generation models to support creator workflows such as dubbing, AI podcasts, and multimodal template creation. The Open Platform exposes APIs for speech synthesis, recognition, and video understanding, with the Pro versions of MOSS‑Transcribe‑Diarize and MOSS‑VL slated for commercial API access.

All models are released under the Apache‑2.0 license, with code repositories on GitHub ( https://github.com/OpenMOSS/MOSS-VL, https://github.com/OpenMOSS/MOSS-Transcribe-Diarize) and model weights hosted on Hugging Face.

In summary, the next frontier for multimodal AI shifts from expanding modality count to building contextual intelligence that can continuously perceive, reason about, and interact with the physical world.

[0.11] [S01] Good morning! [1.03]
[1.11] [S02] Morning, guys! [1.34]