Qwen-Audio-Jev: 20x Faster Audio LLM Inference via Jev Decision Heads

Background: Inefficiency of Autoregressive Audio LLMs

Traditional audio large language models generate textual answers token by token even for simple decision tasks such as speaker verification or audio quality scoring. This autoregressive decoding adds latency and compute overhead.

Qwen-Audio-Jev Architecture

The model builds on Qwen-Audio-3.1-ASR-Flash-Next , which comprises an audio encoder, an audio-to-language adapter, and a 40-layer MoE language decoder (hidden size 2048, 256 routed experts, top-8 activation per token). Instead of redesigning the entire stack, the team inserts a dedicated Jev decision head that consumes the hidden state of the last question-prefix token before answer generation. This state fuses audio content, user question, and answer rules; causal attention prevents peeking at the ground-truth answer during training.

Three Decision Modes

The Jev head contains three linear projection branches sharing the same audio understanding backbone:

CHOICE (multi-class selection): predicts probabilities over 2–64 candidate descriptions via Softmax; candidate order is randomized during training to avoid positional bias. Example: identify whether a sound is dog bark, rain, or music.

NOUL (binary judgment): outputs a single probability via Sigmoid for true/false propositions (e.g., same speaker?). Thresholds can be tuned per business need.

SCORE (ordinal scoring): predicts a distribution over 2–10 ordered levels, computes the expected value, and maps it to an application scale (e.g., 1–5 MOS). Uses a composite loss: cross-entropy (CE) for distribution matching, CDF loss (weight 0.25) for ordinal penalties, and SmoothL1 (weight 0.10) for mean-score alignment. Note: SCORE predicts audio attribute quality, not model confidence.

All three modes avoid token-by-token generation. However, multiple questions still require separate forward passes per question.

Training: Two-Stage SFT + RLCD

Stage 1: Multi-task Supervised Fine-Tuning

Diverse datasets (speaker verification, emotion recognition, audio quality) are unified into an audio-question-answer_rule-target_label format. Each sample activates only its corresponding decision loss. The MoE load-balancing and router z-loss are retained; next-token prediction is dropped.

Stage 2: Reinforcement Learning for Calibrated Decisions (RLCD)

Each decision is framed as a one-step contextual bandit. From the SFT model, 32 samples are drawn with replacement per question, adding 5% uniform exploration. Rewards combine decision correctness and probability distribution quality (expected Brier score); for SCORE an ordered probability score penalizes distant predictions. Policy gradients use importance weighting and a leave-one-out baseline to reduce variance. The goal: accurate decisions and well-calibrated probabilities.

Experimental Evaluation (18 Configurations)

CHOICE: Classification Accuracy

Evaluated on eight benchmarks (acoustic scene, instrument, human non-speech, speaker gender, speech emotion, audio QA). Gender classification and human non-speech sounds exceed 90% accuracy; MELD emotion recognition is weakest. MMSU audio QA reaches >84% accuracy , MMAU follows, MMAR lowest — indicating complex audio QA needs separate improvement. Scene and instrument recognition show room for growth.

NOUL: Binary Judgment

Tested on speaker verification (VoxCeleb1-O), binary audio QA (Clotho-AQA, In-the-Wild), sarcasm detection (PodSarc), and deepfake detection. PodSarc and VoxCeleb1-O achieve AUC >0.98 , EER ~6.5%, FPR/FNR <8% at 0.5 threshold. Clotho-AQA and In-the-Wild have similar AUC (~0.877) and EER (~20%), but In-the-Wild suffers 57.3% FNR vs 24.5% for Clotho-AQA at 0.5 threshold — revealing a threshold calibration issue where false negatives dominate. This separates discriminative ability (AUC) from operating-point trade-offs.

SCORE: Quality Assessment

On NISQA (speech quality dimensions) and VoiceMOS/BVCC (synthesized speech). Overall Quality and Coloration show MAE ~0.3–0.4 and Pearson ~0.8. Loudness has comparable MAE but lower Pearson, indicating mean accuracy ≠ linear correlation. VoiceMOS yields Pearson 0.723 / Spearman 0.754 but MAE 0.813 — high correlation yet large absolute error. Noise dimension has highest MAE (0.584). Key improvement targets: reduce VoiceMOS MAE, improve noise scoring, align loudness predictions with human perception.

Speedup

Across all tasks, Qwen-Audio-Jev achieves ~20× inference acceleration over the base autoregressive model by eliminating token-by-token decoding.