When Does a Song’s Climax Start? GaMMA Lets Multimodal Models Grasp Music Timelines

Problem: Why Existing Multimodal Models Miss Music Timelines

Current multimodal large models can read, see, and hear, yet they fail to understand the temporal structure of music. Global semantic tasks (e.g., genre, instrumentation) ignore time, while temporal tasks (e.g., identifying the chorus segment) require precise time‑axis modeling. Most audio‑multimodal models focus on speech content, leaving melody, harmony, rhythm, and form under‑represented, and they suffer from a single‑encoder conflict between global and temporal representation needs.

GaMMA Core Design

GaMMA introduces a Dual‑encoder Fusion Network (DFN) that separates a Temporal Expert (fine‑tuned on second‑level annotated music structure data) from a Global Expert (learning overall musical semantics). The two experts produce independent audio embeddings that interact through bidirectional cross‑attention and a learnable gate that dynamically balances their contributions at the token level. The fused representation passes through residual connections and a feed‑forward network to generate the final audio embedding.

Three‑Stage Progressive Training

Stage 1 – Multimodal Alignment Pre‑training : Train on millions of music‑text pairs (music‑description and music‑lyrics) with audio clips limited to 60 s and up to 1500 tokens, freezing the LLM backbone and learning a projector from audio to language space.

Stage 2 – Supervised Fine‑tuning (SFT) : Create high‑quality instruction data with second‑level time annotations using SongFormer for segmentation and Gemini 2.5 Pro for generating detailed analysis reports, which are then verified and refined by music experts. The data cover 11 music dimensions and are diversified via GPT‑5.1 paraphrasing. Audio length is extended to 300 s (7500 tokens) and all parameters become trainable.

Stage 3 – Reinforcement Learning (GRPO) : Generate seed samples, estimate solvability with Monte‑Carlo rollouts, filter for medium difficulty (25 % ≤ Pass < 1), synthesize harder question variants with Gemini 2.5 Pro, and finally apply Group Relative Policy Optimization using multi‑choice data. Rewards combine answer correctness and format consistency.

MusicBench: A Large‑Scale Benchmark for Music Temporal Understanding

MusicBench contains 3,739 human‑annotated multiple‑choice questions, split into MusicBench‑Global (2,741 questions covering style, emotion, instrumentation, etc.) and MusicBench‑Temporal (998 questions explicitly testing time‑axis reasoning on vocals, instruments, structure, chords, and lyrics).

Experimental Results

On MuChoMusic, GaMMA‑8B achieves 78.0 % overall accuracy, outperforming Kimi‑Audio (68.2 %) and Audio‑Flamingo3 (73.4 %). After upgrading the base model to Qwen3‑14B, accuracy rises to 79.0 %.

On MusicBench‑Global, GaMMA‑8B scores 82.6 %, surpassing Gemini‑3.0 Pro’s 80.4 %. On MusicBench‑Temporal, GaMMA‑14B reaches 75.0 % on chord understanding (vs. 53.0 % for Gemini‑3.0 Pro) and shows strong results on structure analysis (86.5 %) and lyric alignment (95.5 %). Human expert evaluation of open‑ended generation further confirms GaMMA‑14B’s dominant performance.

Qualitative Demo: Multi‑Turn Music Dialogue

GaMMA can answer user queries such as locating the emotional climax (e.g., “around 2:35 to 3:32”) and providing detailed structural analysis, including time ranges, instrument lists, chord progressions, time signatures, rhythmic feel, and melodic motifs for each section of a song.

Conclusion

GaMMA demonstrates that a dedicated dual‑encoder architecture combined with progressive, time‑aware training can unify global and temporal music understanding within a single parameter set. MusicBench offers the most comprehensive evaluation of music LMMs to date, and GaMMA’s superior results suggest a significant step toward truly multimodal AI that can comprehend music as humans do.