CAPO-SLT: Confidence-Aware RL Fixes Fluent-but-Wrong Sign Language Translation

Automatic sign language translation (SLT) converts continuous video or pose sequences into natural language, bridging communication for deaf and hard-of-hearing users in public services, healthcare, education, and daily life. However, a hidden problem persists: even when visual-text alignment is strong, autoregressive generation can be derailed by locally plausible tokens that lack visual evidence. These errors accumulate as context for subsequent tokens, causing whole-sentence semantic drift — fluent but wrong translations that are harder to detect than obviously broken output.

Why Alignment Is Not Enough

SLT faces modality gaps, temporal granularity mismatches, and structural differences between sign and spoken languages. Recent work focuses on visual-language alignment, but alignment does not guarantee stable generation. Strong language priors make some continuations sound natural; without sufficient visual support, fluency masks errors. Reinforcement learning (RL) can optimize for sentence-level quality, but rewards are assigned at the sequence level while advantages must be distributed per token. The model knows the whole sentence is better or worse, yet cannot pinpoint which tokens caused the shift.

The Core Issue: One Clipping Bound for All Tokens

Policy optimization methods like PPO and GRPO clip the probability ratio between new and old policies to prevent overly large updates. Traditionally, all positive-advantage tokens share a fixed clipping range (e.g., 0.2). This ignores that tokens are in vastly different states:

Tokens where the old policy already has high confidence may simply reflect language priors — "easy" continuations — and further amplifying their probability reinforces wrong paths. Tokens with low old-policy probability but positive advantage may be precisely the corrections worth exploring, yet they are constrained by the same ceiling.

CAPO (Confidence-Aware Policy Optimization) treats old-policy probability not as proof of correctness but as a signal for how much update magnitude is appropriate.

CAPO-SLT: Dynamic Per-Token Clipping

For positive-advantage tokens, CAPO sets the upper clipping bound as a function of the old policy's confidence. Higher confidence → bound closer to the conservative base (0.2); lower confidence → bound relaxed up to a maximum of 0.3. This is not suppression: high-confidence tokens with positive advantage are still reinforced, just not over-amplified. For negative-advantage tokens, CAPO adds an upper bound on loss magnitude, reducing the risk that sentence-level reward noise excessively penalizes individual tokens.

The training pipeline:

Supervised initialization: Full-body pose keypoints extracted from video, split into face, body, left hand, right hand. ST-GCN models spatial joint relations and temporal motion, learning shared visual-text semantic representations.

RL fine-tuning: Multiple candidate translations sampled per sign segment. Sentence-level reward combines BLEU-1, BLEU-4, ROUGE-L, and BERTScore; group-relative advantages computed.

Confidence-aware update: Dynamic clipping for positive-advantage tokens based on old-policy probability; loss clipping for negative-advantage tokens. Visual backbone and reward function unchanged.

Chinese side: large-scale pre-training on CSL-News, supervised fine-tuning and RL on CSL-Daily. English side: pre-training on YouTube-ASL, evaluation on How2Sign. Isolated sign recognition tested on WLASL2000.

Results: Pose-Only, Top Scores Across Chinese Benchmarks

On CSL-Daily test set, compared to prior pose-only methods:

vs. Geo-Sign: +2.93 BLEU-1, +1.26 BLEU-4, +0.64 ROUGE-L

vs. Uni-Sign: +4.96 BLEU-1, +3.07 BLEU-4, +3.67 ROUGE-L

CAPO-SLT achieves the highest values in all three metrics among pose-only entries, even outperforming some methods that also use RGB frames.

Cross-language and auxiliary tasks:

How2Sign (ASL, pose-only): 41.4 BLEU-1, 15.2 BLEU-4, 34.9 ROUGE-L — improvements of +1.0, +0.7, +0.6 over pose-only Uni-Sign.

WLASL2000 isolated recognition: 63.77% Per-Instance, 61.91% Per-Class accuracy, showing pose representations remain competitive for sign understanding.

Ablation: Where Gains Come From

SFT, GRPO, DAPO, and CAPO share the same supervised checkpoint, reward, and training setup; only the policy optimization rule differs. Compared to DAPO, CAPO improves BLEU-1, BLEU-4, ROUGE-L by +0.30, +0.50, +0.67 respectively. Removing the negative-advantage cap drops scores to 58.48/28.03/57.80. Replacing the inverse-confidence mapping with a linear mapping of the same range reduces BLEU-4 by 0.73 and ROUGE-L by 0.60.

Qualitative Examples: Fluency vs. Fidelity

Without CAPO, the model produces fluent but semantically flipped translations:

Reference: "I introduce these colleagues separately" → Model: "I am a team composed of several people"

Reference: "It's dark, I'm scared" → Model: "It's dark, I fell asleep"

Reference: "The actress in this movie is beautiful" → Model: "This movie is about the actress's love story"

All three errors replace predicates, entity relations, or event meanings while maintaining surface fluency. With CAPO, all three revert to reference-equivalent or semantically equivalent outputs. Remaining errors are local omissions, tense shifts, or minor phrasing differences — not whole-sentence meaning reversals.

Training Dynamics: Entropy Stability

GRPO's average entropy drops quickly then rebounds in later training, indicating renewed policy uncertainty. CAPO's entropy declines early then stabilizes without sudden collapse or late rebound, confirming that confidence-aware clipping yields more controlled token-level updates, matching the goal of stable generation.

Portability and Limitations

CAPO's design is engineering-friendly: it modifies only the clipping rule in the policy optimization step, leaving the visual backbone and reward function untouched. For existing SLT systems with supervised initialization and PPO/GRPO pipelines, it offers a low-intrusion stabilization path.

Limitations: Gains on ASL (How2Sign) are more modest than on CSL-Daily. Current rewards rely entirely on reference translations; n-gram overlap metrics (BLEU, ROUGE) combined with BERTScore still miss acceptable paraphrases, discourse-level adequacy, and fine-grained semantic preferences. Future work will explore learned preference models or SLT-specific semantic evaluators.

Paper: CAPO-SLT: Confidence-Aware Policy Optimization for Stable Sign Language Translation Generation (EMNLP 2026). Authors: Ling Zhou, Yuhao Chen, Lin Cheng, Chengwen Yao, Donghui Sun, Xiaoxin Chen. OpenReview: https://openreview.net/pdf?id=l4bCsrSkYx