AI RAP: End-to-End Speech Synthesis for Rap Generation Using Location‑Sensitive Attention and Inference Mask

Background: The popular Chinese rap show "China New Rap" attracted massive attention, generating a strong desire among users to create freestyle rap but lacking an easy creation environment.

To address this, the team released an AI‑powered service called "AI RAP" that enables ordinary users to generate personalized rap songs with a single click, fulfilling the vision of making entertainment simpler and more fun.

Technical challenges: Compared with normal speech, rap features fast tempo and blurred phoneme boundaries. Conventional end‑to‑end TTS models can produce continuous audio but struggle to obtain precise character boundaries and suffer from alignment accuracy and real‑time performance issues. AI RAP implements several optimizations to overcome these problems.

AI RAP integrates multiple AI techniques. It is the first system to combine location‑sensitive attention with an inference mask, significantly improving alignment correctness. Based on the alignment results, the synthesized speech is automatically matched to the beats of background music, and an optimal stretching strategy is applied to align each word with the corresponding rap tempo, producing a professional‑level flow.

In the speech synthesis component, while Google’s Tacotron model restores timbre well, it often yields unstable monotonic alignments, leading to missing or repeated characters. AI RAP further refines the Tacotron architecture, markedly enhancing alignment accuracy and synthesis quality, and supports multiple voice timbres.

The system adopts the location‑sensitive attention module from Tacotron‑2 and introduces an inference mask during inference. This mask limits attention to maintain monotonicity, raising alignment accuracy on an 8,000‑plus test set from 80 % to 100 %.

Attention optimization results (before vs. after) demonstrate a clear improvement in alignment stability and naturalness.

Voice timbre: By training on a large dataset, AI RAP can generate IP‑character voices such as "Crayon Shin‑chan", "Chibi Maruko‑chan", and popular comedy personalities, satisfying diverse age groups and enabling users to have celebrities “sing” their lyrics.

Synthesis speed: The system replaces the iterative Griffin‑Lim algorithm with a proprietary waveform reconstruction algorithm, achieving synthesis of up to 40 characters in about one second on a CPU.

Scalability: With the speed optimizations and a distributed deployment architecture, AI RAP can support millions of daily users, allowing widespread rap creation and sharing.