Index-Translate: Training a 150-Language Multimodal Translation Model Family

1 Overview

Bilibili's Index LLM team releases the Index-Translate series: text translation covering 150 languages in three sizes (2B, 9B, 35B-A3B preview), with image-text translation, speech translation, syllable control, and long-document translation capabilities. The models are open-sourced; GitHub stars exceed 1,600 and Hugging Face downloads surpass 100k within two weeks.

The core model Index-Translate handles multilingual text and image-text translation, following terminology, format, style, and context instructions. Three derivative models address specific constraints: Index-Echo for speech-to-text and speech-to-speech translation with source-audio timbre conditioning; Index-Homura adjusts translation length to a given syllable budget for subtitling and dubbing; Index-NativeLong translates full documents in one pass, learning cross-paragraph consistency.

Online Demo: https://index-translate.bilibili.com/

GitHub: https://github.com/bilibili/Index-Translate

Hugging Face: https://huggingface.co/collections/IndexTeam/index-translate

Free API (OpenAI-compatible): https://github.com/bilibili/Index-Translate#option-1-free-online-api-zero-gpu-setup

Full method and experiments are in the technical report (arXiv:2609.40181) [1].

2 Multilingual Base Formation

Index-Translate builds on Qwen3.5 base [2]. Mid-training uses a mix of general text replay (G), independently sampled monolingual text (M), and parallel translation data (P); G and M preserve language understanding and generation, while P strengthens cross-lingual alignment. Later, pivot data (V) linking multiple language versions of the same content is added [3].

The team adopts the constant and decay phases of the WSD (Warmup–Stable–Decay) learning-rate schedule [4]. Constant phase keeps a fixed learning rate with balanced G, P, M proportions; decay phase gradually lowers the rate and uses pivot data as the main component while maintaining core and full language coverage. All three sizes share the same base text recipe, using AdamW, 4,096-token sequence length, and 4,096-sequence global batch.

Table 1: Base Text Model WSD Mid-training Recipe (see image). Pivot-20 and pivot-148 counts exclude English and Chinese; adding them yields 22 core language entries and 150 full language entries. Pivot is a training data organization; inference directly generates the target language.

Recipe selection references pilot experiments in the technical report. Table 2 shows representative constant and decay configurations under equal decay budget (25.17B token positions). Constant phase evaluated at step 7,000 (4K context) on FLORES-200 and WMT24++ English/Chinese source COMET averages over 20/18 target languages. Decay phase uses Qwen3-8B, 16K context, step 10,000, evaluating 20 target languages. NLU is macro-average over 11 multilingual tasks.

Key findings:

Constant phase needs both monolingual and parallel data. The 1:1:1 ratio (G:M:P) achieves highest translation score and joint-highest NLU; removing either monolingual or parallel data lowers both scores. Increasing parallel weight to 1:1:4 does not improve translation further.

Decay phase "pure pivot" (G+M+V, no ordinary parallel P) yields translation score 0.7991 vs. 0.7971 for parallel+pivot (G:M:P:V = 1:2:2:4); the latter has slightly higher NLU. Adding P changes normalized shares of other streams, so this compares two full recipes.

Phase gains tracked on 2B and 9B: 2B translation score progresses 0.7015 → 0.7707 → 0.7898 (base → constant end → decay end); 9B progresses 0.7726 → 0.7924 → 0.8007. Both phases show translation gains. NLU drops in constant phase, partially recovers in decay, but remains below base. Results from 20+ language dev sets.

Language coverage verified separately: expanding decay data from 20+ to 100+ languages on 2B raises corrected COMET average on 61 non-core languages from 0.5605 to 0.6570 (wrong target language outputs counted as zero) and reduces target language deviation from 1.94% to 0.89%; core language averages stay stable. Post-training continues low-resource emphasis to sustain coverage.

3 Learning Usable Translation via Task Rewards

3.1 Learn from Examples Then Optimize Task Objectives

On the shared base, three experts are trained separately: general translation, instruction translation, and meme translation. Supervised fine-tuning (SFT) provides high-quality examples; reinforcement learning (RL) further optimizes toward task-specific objectives.

General translation must avoid omissions, additions, and factual shifts. Semantic similarity metrics alone may not sufficiently penalize these errors. The reward combines target-language validity gating, reference-based XCOMET-XXL quality score [17], and source-based faithfulness score. Language errors trigger gating; faithfulness errors reduce reward, preventing fluent but incomplete translations from receiving high scores.

Instruction translation handles concrete requirements (e.g., "keep JSON structure", "translate product names per glossary", "preserve code placeholders", "use a specific narrative tone"). Rubric-as-Reward breaks constraints into checkable hard conditions and graded soft conditions. Hard checks (structure, terminology, placeholders) failing zero the reward via gating; soft conditions (style, context understanding) contribute to graded scoring.

General and instruction experts use GRPO [1] optimizing via relative rewards among multiple candidates per input. Meme translation uses context- and reference-guided review to learn non-literal meaning, cultural function, and pragmatic effect, choosing free translation, explanation, or cultural adaptation when needed.

3.2 Keep the Critic Aligned with Model Changes

Reward functions can be gamed: higher scores may not reflect true quality improvement. As the policy model updates, the output distribution shifts, challenging a fixed critic.

The team adopts RIVAL's adversarial critic supervision [5], alternating critic and policy updates, combined with preference ranking, auxiliary quantitative supervision, and historical output replay. This lets the critic continuously learn the current policy's error patterns, making RL supervision adapt to model behavior and reducing reliance on a static critic's weaknesses.

3.3 Improve Weak Task Performance After Expert Merging

Different experts excel at different tasks, but merging parameters does not guarantee all capabilities are retained. Compatible experts are first merged via parameter interpolation [6] (weights 0.8/0.1/0.1 for general, instruction, meme experts in the 2B and 9B configs). The merged model is then evaluated to identify remaining weak tasks.

Targeted multi-teacher on-policy distillation (MOPD) [7] follows: the student generates its own translation, then the task-matched teacher provides token-level supervision using a reverse-KL objective. The student learns on its actual output distribution to improve post-merge weaknesses; effectiveness is assessed jointly on translation quality and constraint adherence.

This pipeline connects "learn each task well", "integrate complementary capabilities", and "check integration gaps", while preserving observation of capability transfer boundaries. For example, precise syllable control transferred weakly under the studied merge and distillation setup, so the dedicated Homura model is retained.

4 How Index-Echo Connects Translation and Speech

4.1 From Audio Understanding to Target Language Text

Index-Echo's speech-to-text translation (S2TT) comprises an audio encoder, a connector, and the Index-Translate decoder. The audio encoder uses Qwen3-Omni's AuT module [8]; the connector feeds audio features into the translation model; target language instructions control output direction. Training updates these modules end-to-end via speech translation supervision, enabling the text base to directly leverage source audio for translation.

4.2 Connect Speech Generation via Hidden States

Speech-to-speech translation (S2ST) extends the trained S2TT. A ~30M parameter Hidden2CV mapper projects the translation model's final-layer hidden states to CosyVoice3's semantic layer [9], letting the speech generation path receive the translation model's continuous representations. Source audio simultaneously provides a reference prompt and CampPlus speaker embedding for cross-lingual timbre conditioning.

Training proceeds in two steps: first, freeze S2TT and speech generation backbones, distill to align the mapper; then apply differentiable reward optimization (DiffRO), keeping S2TT frozen. DiffRO uses Gumbel-Softmax to approximate speech token sampling differentiably, allowing a Token2Text content consistency reward to backpropagate through the speech generation path, optimizing generated content alignment with the target translation [9].

Released S2ST covers six directions: Chinese → English, Spanish, Japanese; English → Chinese, Spanish, Japanese. The demo shows video subtitle translation and cross-lingual dubbing.

5 Specialized Training for Time Budget and Long Documents

5.1 Homura Brings Syllable Budget into Training Objective

In dubbing, a semantically correct translation may not fit the original video's time window. Simply asking for "shorter" output is unstable; over-compression loses information.

Index-Homura takes target syllable count as explicit input and uses GRPO to jointly optimize translation quality and syllable deviation. The model adjusts expression length via vocabulary selection and syntactic restructuring to approach the given budget while preserving meaning. Syllable count provides an actionable control knob for time-constrained translation; final playback duration also depends on speech rate, pauses, and TTS settings.

In the technical report's SandGlass evaluation, Index-Homura-9B achieves 81.92% of translations within ±10% of the target syllable count. This evaluation simultaneously checks translation quality and budget adherence, enabling analysis of length-control gains versus content loss. HOMURA background and method detailed in arXiv:2601.10187 [10].

5.2 NativeLong Learns Continuous Full-Document Generation

Document translation difficulty evolves with context length. Character names, terms, and concept distinctions may reappear dozens of paragraphs later; chunk-wise translation yields locally fluent but globally inconsistent output (e.g., drifting honorifics, concept confusion).

Index-NativeLong uses full source and full target text as document-level supervision, training the model to generate continuously while retaining full-document context [11]. The 2B model combines long-sequence mid-training decay with document SFT; the 9B model continues document training from the merged text model. Training and evaluation cover document lengths of approximately 4K–64K Chinese-side tokens.

Example: a geological text first distinguishes "ice age" and its "glacial" stages; later occurrences must consistently preserve the "ice age" vs. "glacial" distinction. Document-level training targets such cross-paragraph relations, as well as omissions, premature stopping, and terminology inconsistency in long outputs. The team also built NativeLongBench, evaluating full-document translation across length buckets.

6 Evaluating Different Capabilities with Different Evaluations

6.1 Text Translation Evaluation

General translation, instruction following, cultural expression, and document consistency require separate benchmarks. The team uses FLORES [12] and WMT24++ [13] for general quality; instTrans for format, terminology, style, and context constraints; MEME, SandGlass, and NativeLongBench for respective specialized capabilities.

In the report's text comparison, Index-Translate-35B-A3B (preview) achieves FLORES COMET-22 0.8794 and instTrans IFscore 0.8336. All text evaluation baselines disable thinking mode; speech baselines with thinking are explicitly marked. Full settings, results, and analysis in the technical report (arXiv:2609.40181).

Table 3a: General Translation, Instruction & Cultural Expression (see image). FLORES uses COMET-22 [16]; WMT26 uses 0–100 reviewer scores; other metrics 0–1, higher better. instTrans quality and IFscore judged by GPT-5.6-Sol; MEME evaluates community and cultural expression translation quality. Bold indicates column best.

Table 3b: WMT24++, IFMTBench & Vertical Evaluations (see image). All metrics 0–1, higher better. IFMTBench [14] reports translation quality and instruction following separately; vertical average is equal-weight mean of five domain translation scores, including GuoFeng literary translation [15].

Table 4a: Low-Resource General Translation FLORES_minor_pair (see image). Table 4b: Low-Resource Instruction Translation instTrans_minor (see image). FLORES_minor_pair covers 1,040 directions across 62 languages; instTrans_minor contains 2,793 instruction tasks. Deviation rate = proportion of outputs not in target language (lower better); other metrics higher better. North-Small-Translate refers to 218B-A25B model; TranslateGemma-12B refers to translategemma-12b-it. Full evaluation table at https://github.com/bilibili/Index-Translate/blob/main/docs/evaluation.md .

6.2 Emergent Image Translation Capability

Index-Translate exhibits image translation ability after text-only training: it can directly read text in images and generate target language translations. The model inherits Qwen3.5's vision pathway; both mid-training and post-training use only text data [1,2].

Three evaluation sets: MMTIT-Bench [18] (1,243 images, 14 source languages → English/Chinese); FLORES-rendered (FLORES_minor_pair text rendered as 1,200 solid-background images, 60 source languages → two targets); Vistra [19] (763 natural-scene images, four target languages, reference translations concatenated in reading order for whole-image evaluation). Zero-shot image translation instruction, COMET-22 [16] for quality, unified non-thinking inference.

Index-Translate-35B-A3B (preview) achieves highest scores in all three: MMTIT-Bench 0.7845, FLORES-rendered 0.7050, Vistra (adapted) 0.7168, outperforming same-size Qwen3.5 by +0.0038, +0.0124, +0.0137 respectively. Scores improve stepwise from 2B to 9B to 35B-A3B, showing stronger cross-modal translation with scale.

Table 5: Image Text Translation Evaluation (see image).

Next version will further unify speech translation, vision-language translation, and text translation, incorporating long-text translation into a single model. Unified multilingual and multimodal capabilities will enable one model to handle text, image, and speech inputs for translation, while strengthening cross-paragraph context linking, terminology consistency, and complete translation generation for long-form content, preserving original semantics and expressive style.

7 Online Demo and Free API

Try the demo at https://index-translate.bilibili.com/ ; inference code and tutorials on GitHub https://github.com/bilibili/Index-Translate ; model repositories on Hugging Face https://huggingface.co/collections/IndexTeam/index-translate .

A free public API requiring no local GPU is provided, compatible with OpenAI chat completions and responses endpoints. Current served model identifier: Index-Translate-35B-A3B . Example curl request for Chinese translation:

curl https://index-translate.bilibili.com/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
  "model": "Index-Translate-35B-A3B",
  "messages": [
    {
      "role": "user",
      "content": "Translate into Chinese: Welcome to EMNLP."
    }
  ],
  "temperature": 0
}'

API base URL: https://index-translate.bilibili.com/v1 . More usage examples in the free API docs https://github.com/bilibili/Index-Translate#option-1-free-online-api-zero-gpu-setup . The team welcomes feedback on GitHub with custom glossaries, subtitles, novels, or documents.

8 EMNLP 2026 On-site

If attending EMNLP 2026, join the team's oral presentation, poster sessions, and WMT poster session to discuss translation, RL, and multilingual content production.

HOMURA oral : 2026-10-27 09:00–10:30, Machine Translation and Semantics session, room F9-10.

Poster Session D : 2026-10-26 11:00–12:30, Hall H.

WMT poster session : schedule per WMT official program.

Times are Budapest local time; confirm via EMNLP official schedule https://2026.emnlp.org/program/ and WMT schedule https://www2.statmt.org/wmt26/program.html . The team will also distribute merchandise; welcome to exchange technical insights and take home goodies.

References

[1] Tianjiao Li et al. Index-Translate: A Multilingual Translation Model Family — Text, Speech, Controlled Dubbing, and Long-Document Translation. arXiv:2609.40181, 2026. https://arxiv.org/abs/2609.40181

[2] Qwen Team. Qwen3.5: Towards Native Multimodal Agents. February 2026. https://qwen.ai/blog?id=qwen3.5

[3] Yingli Shen et al. From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corpora. EMNLP 2025, pp. 7357–7379. https://aclanthology.org/2025.emnlp-main.374/

[4] Shengding Hu et al. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies. arXiv:2404.06395, 2024. https://arxiv.org/abs/2404.06395

[5] Tianjiao Li et al. RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation. arXiv:2506.05070, 2025. https://arxiv.org/abs/2506.05070

[6] Mitchell Wortsman et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. ICML 2022, PMLR 162:23965–23998. https://proceedings.mlr.press/v162/wortsman22a.html

[7] Wenhan Ma et al. MOPD: Multi-Teacher On-Policy Distillation for Capability Integration in LLM Post-Training. arXiv:2606.30406, 2026. https://arxiv.org/abs/2606.30406

[8] Jin Xu et al. Qwen3-Omni Technical Report. arXiv:2509.17765, 2025. https://arxiv.org/abs/2509.17765

[9] Zhihao Du et al. CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training. arXiv:2505.17589, 2025. https://arxiv.org/abs/2505.17589

[10] Ziang Cui et al. HOMURA: Taming the Sand-Glass for Time-Constrained LLM Translation via Reinforcement Learning. arXiv:2601.10187, 2026. https://arxiv.org/abs/2601.10187

[11] Yushi Bai et al. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs. ICLR 2025. https://arxiv.org/abs/2408.07055

[12] NLLB Team et al. No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv:2207.04672, 2022. https://arxiv.org/abs/2207.04672

[13] Daniel Deutsch et al. WMT24++: Expanding the Language Coverage of WMT24 to 55 Languages & Dialects. Findings of ACL 2025, pp. 12257–12284. https://aclanthology.org/2025.findings-acl.634/

[14] Mao Zheng et al. Hy-MT2: A Family of Fast, Efficient and Powerful Multilingual Translation Models in the Wild. arXiv:2605.22064, 2026. https://arxiv.org/abs/2605.22064

[15] Longyue Wang et al. Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation. WMT 2023, pp. 55–67. https://aclanthology.org/2023.wmt-1.3/

[16] Ricardo Rei et al. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. WMT 2022, pp. 578–585. https://aclanthology.org/2022.wmt-1.52/

[17] Nuno M. Guerreiro et al. xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detection. TACL 12:979–995, 2024. https://aclanthology.org/2024.tacl-1.54/

[18] Gengluo Li et al. MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation. CVPR 2026, pp. 16593–16602. https://openaccess.thecvf.com/content/CVPR2026/html/Li_MMTIT-Bench_A_Multilingual_and_Multi-Scenario_Benchmark_with_Cognition-Perception-Reasoning_Guided_Text-Image_CVPR_2026_paper.html

[19] Elizabeth Salesky, Philipp Koehn, Matt Post. Benchmarking Visually-Situated Translation of Text in Natural Images. WMT 2024, pp. 1167–1182. https://aclanthology.org/2024.wmt-1.115/