ControlAudio Enables Scripted Timing and Speech Control in Text-to-Audio Generation

Recent Text-to-Audio (TTA) research has achieved high‑fidelity synthesis but still struggles with fine‑grained control, especially precise timing of sound events and the intelligibility of generated speech.

Core Method

The authors propose ControlAudio , a progressive diffusion approach that jointly models text , timing , and phoneme conditions. The method consists of three parts:

Data construction and representation: a hybrid pipeline of manual annotation and simulation creates multi‑level data; structured prompts encode text, timing, and phoneme information for a pretrained text encoder.

Model training: a progressive strategy first pre‑trains on large text‑audio corpora, then fine‑tunes with timing data, and finally adds phoneme conditioning to enable finer control.

Guided sampling: during inference, early diffusion steps use only text and timing to generate the overall temporal structure, while later steps introduce phoneme cues with stronger guidance to refine speech content.

Progressive Diffusion Modeling

Training proceeds in three stages: (1) text‑audio pre‑training learns basic generation; (2) timing‑annotated fine‑tuning teaches the model to align sound events to specified intervals; (3) phoneme‑level joint training equips the model to produce understandable speech. The inference sampler mirrors this coarse‑to‑fine process, first producing a temporal skeleton and then enriching it with detailed phonetic content.

Dataset Construction

To address the scarcity of time‑annotated audio, the team builds a mixed data system. Real data are derived from AudioSet‑SL with timestamps and transcriptions, expanding <text, audio> to <text, timing, phoneme, audio>. Simulated data are generated by statistically modeling speech activity, synthesizing single‑ or multi‑speaker segments, arranging them according to plausible timelines, and mixing with background sounds, resulting in over 170,000 training samples. Structured prompts are automatically created using a Chain‑of‑Thought (CoT) pipeline that parses natural language into “event — time — speech” triples.

Experimental Results

Evaluation on the AudioCondition benchmark shows that ControlAudio markedly improves event‑time alignment while maintaining or surpassing baseline scores on FAD and CLAP audio‑quality metrics. In speech‑focused tests, the model delivers clearer, more understandable speech and retains the high‑fidelity generation of standard TTA tasks.

Conclusion and Outlook

By addressing data, training, and sampling, ControlAudio solves the fine‑grained control problem in text‑to‑audio generation, offering stronger generality and extensibility than prior single‑dimension approaches. The authors anticipate that the “multi‑granularity conditional modeling + progressive generation” paradigm will guide future research toward unified, controllable audio, speech, and music synthesis.