Building a Universal Sound Model: Xiaomi Dasheng’s 8‑GPU AI Engineering Journey

Technical Choice: MAE‑Based Audio Pre‑training

Xiaomi Dasheng starts from the observation that existing audio research focuses on speech recognition (ASR) and lacks a unified representation for speech, environmental sounds, and music. To fill this gap, the team adopts Meta’s Masked Autoencoder (MAE) framework, masking portions of the audio spectrogram and forcing the model to reconstruct them, thereby learning intrinsic audio structures rather than task‑specific features.

Incremental optimization on the ASR path cannot address the need for a universal sound representation; a fundamental route change is required.

Data Engineering: Massive Audio Selection and Scaling

After fixing the algorithm, the next challenge is data. Public video sources are 80‑90% speech; pure environmental sounds and music are scarce. The team therefore uses the largest public audio dataset available at the time, covering roughly 1,000 years of open‑source recordings. By synchronizing video‑audio signals, they filter meaningful clips (e.g., a dog’s bark aligned with a visual dog).

The raw data volume reaches ~300 TB (equivalent to millions of hours of audio). The dataset is split into 146 packages covering speech, music, ambient sounds, and mechanical noise. Using a multi‑machine, staggered download/upload pipeline, the team moves the data over roughly one year. With a single 8‑GPU machine they train models ranging from a 86 M‑parameter “Base” to a 1.2 B‑parameter model.

Scaling results on the HEAR benchmark show average performance rising from 78.88 (Base) to 81.25 (1.2 B). Expanding training data from AudioSet’s 5,000 h to 270,000 h yields additional gains of 6.37, 8.69, and 8.45 percentage points for the three model sizes respectively, confirming that “larger model + more data” also benefits audio.

However, a ten‑fold data expansion initially caused performance drops on the public AudioSet test set, revealing that data quality outweighs sheer quantity and that open‑source metrics correlate strongly with real‑world business metrics.

Semantic Expansion: Six‑Dimensional Annotation for Natural Audio Description

While the model can now understand sounds, its output is limited to class probabilities, which lack semantic richness (e.g., it can signal “wind” but not describe “strong wind with distant speech”). To enable textual output, the team builds a six‑dimensional caption dataset (ACAVCaps) covering speech content, speaker emotion, background sound, music, scene environment, and audio type, and releases the accompanying MECAT benchmark.

Initially, the team doubted the value of multi‑dimensional labeling, fearing redundancy. Experiments later proved that fine‑grained six‑dimensional annotations are crucial for generating realistic acoustic scenes.

Architectural Breakthrough: Unifying Understanding and Generation

Previously, understanding and generation relied on separate encoders and decoders, doubling compute and deployment cost. The team proposes a unified architecture where a single encoder (the Dasheng model) is frozen for semantic features while a lightweight layer injects acoustic information, enabling the same model to both “listen” and “speak”.

Experiments show that the new DashengTokenizer surpasses prior audio encoders and audio‑codec pipelines on 22 tasks, including text‑to‑audio, text‑to‑music, and speech enhancement, outperforming standard VAE methods.

Challenging industry assumptions is not about disproving them but about discovering their boundaries; DashengTokenizer demonstrates that VAE is not the only solution.

Future Exploration: Scene‑Aware Audio Generation

The roadmap continues from “giving devices ears” to “giving devices mouths”. The upcoming project “DashengAudioGen” aims to generate full acoustic scenes—including environmental sounds, reverberation, and spatial cues—rather than clean speech alone.

All code, data, and models are open‑sourced, with links to the GitHub repository and demo web page.