How Doubao’s Video Calls Got Faster and Smarter with Volcano Engine’s Multimodal Transmission

Doubao’s video‑call feature has been upgraded so that AI can interact naturally in continuously changing real‑world scenes, delivering three immediate user benefits: simultaneous audio, video and text processing, proactive AI prompts, and a natural, interruption‑free dialogue rhythm.

1. Seconds to connect, seconds to answer

Traditional RTC separates media and signaling channels, requiring multiple negotiation rounds that add several seconds before a call becomes usable. The new Volcano Engine Multimodal Transmission (MMT) unifies media and model sessions using a QUIC‑based client and the MoQ protocol, allowing a single connection to carry audio, video, signaling and model state together.

Client built on QUIC library – multiplexes all streams over one connection.

Transport layer uses MoQ protocol – provides unified session control and coordinated scheduling of media and control signals.

As a result, connection latency is compressed from seconds to a few hundred milliseconds, making the call feel instantly responsive.

2. Zero dropped words, no answer‑mismatch

Previously, audio and model inference channels were established asynchronously, causing situations where the model received incomplete or misaligned input, leading to “dropped words” or irrelevant answers. MMT solves this with a unified multimodal session controller that synchronizes audio‑video streams and model state on the same transport link.

MediaKit algorithm at the gateway checks first‑frame completeness, audio‑video alignment, and model readiness before triggering inference.

Only after all modalities are synchronized does the model start processing, eliminating missing‑word and off‑topic responses.

Latency jitter exceeding one second, a known source of input distortion, is systematically reduced under MMT’s fine‑grained session control.

3. Precise intent understanding and intelligent inference

AI interaction requires more than raw transport; the system must decide when to request high‑resolution images, extract frames, or adjust bitrate based on the user’s query. MMT’s client‑server design turns the transport layer into an “intelligent scheduling layer”.

Gateway makes key decisions – routes raw audio/video directly to the model or applies preprocessing such as frame extraction or high‑resolution capture.

Layered session control – MoQ signaling defines priority of audio streams, selects which video frames to forward, and balances reliability versus low‑latency delivery.

Multimodal synchronization – voice, visual, and temporal information are aligned before reaching the model, ensuring the model receives a coherent, packaged input.

This approach lets the transmission system “understand” the model’s needs and deliver the right data at the right time, rather than merely shuttling all data indiscriminately.

4. Model sets the AI ceiling, transmission sets the experience floor

The competition in real‑time AI interaction is not only about smarter models; the underlying transport infrastructure must convey model capabilities without loss. By upgrading the transmission stack from a “dumb pipe” to an “intelligent scheduling layer”, Volcano Engine enables large‑scale, low‑latency delivery of AI power to every end‑user. Doubao’s integration of the native full‑duplex large model SeedRealtime showcases this end‑to‑end capability.

As scenarios such as simultaneous translation, language tutoring, and museum guides accelerate, “watch‑listen‑speak” multimodal interaction will become a standard experience, underpinned by a transmission system validated by billions of users and capable of handling hundred‑fold complexity.