How Make-An-Audio Turns Text Into Realistic Sound Effects

Make-An-Audio is a new text‑to‑audio generation model jointly developed by Zhejiang University, Peking University and Volcano Speech, capable of producing realistic sound effects from natural language descriptions and supporting any modality (text, audio, image, video) as input.

Recent AIGC trends have highlighted breakthroughs such as generating images, video and 3D models from text; however, audio synthesis has lagged due to scarce paired data and long‑duration waveform modeling. Make‑An‑Audio addresses these challenges with a “Distill‑then‑Reprogram” text‑augmentation strategy that first distills natural language descriptions from a teacher model and then reprograms them with randomly recombined event data to create dynamic training samples.

Key technical components include a self‑supervised audio‑to‑text and audio‑text retrieval system for the Distill stage, a latent diffusion model for predicting spectrogram representations, and CLAP‑based contrastive language‑audio pretraining together with large language models (T5, BERT) for robust text conditioning. The model also introduces a CLAP Score for evaluating audio‑text consistency and demonstrates strong zero‑shot generalization on benchmark datasets.

Demonstrations show the model generating audio for prompts such as “a speedboat running as wind blows into a microphone” and “fireworks pop and explode”, as well as repairing damaged audio and converting images or video frames into corresponding sound effects. Sample videos have attracted tens of thousands of views on social media.

Make‑An‑Audio embodies the “No Modality Left Behind” principle, enabling high‑quality, controllable audio synthesis from any input modality, and is expected to impact film dubbing, short‑video creation and broader AIGC applications, though occasional mismatches between text and generated audio remain.

Paper link: https://arxiv.org/abs/2301.12661 Project link: https://text-to-audio.github.io