Speech Translation: Enterprise Applications and Research

Overview

Speech translation converts spoken language in one language into text (or speech) in another language. It aims to break language barriers for communication, cultural exchange, and information dissemination, with applications such as automatic subtitles on video platforms, real‑time interpretation in meetings, and translation devices for travel.

Enterprise Applications at ByteDance

ByteDance leverages its translation platform (Volcano Translation) to support internal communication across its global workforce and to provide multilingual subtitles for user‑generated videos. Recent products include AR smart‑translation glasses that offer real‑time subtitles, face‑to‑face translation, and photo translation for travel scenarios.

1. Modeling Methods

Cascade Speech Translation

Traditional systems chain an automatic speech recognition (ASR) model with a machine translation (MT) model. This modular approach benefits from large‑scale ASR and MT data but suffers from error propagation, computational complexity, and the need for additional error‑handling modules.

End‑to‑End Speech Translation

End‑to‑end models directly map audio to target‑language text using an encoder‑decoder framework, often based on the Transformer. They mitigate error propagation and simplify deployment but face data scarcity challenges.

2. Better End‑to‑End Models

Encoder Improvements

LUT (Listen‑Understand‑Translate, AAAI 2021): adds a semantic encoder supervised by ASR transcripts and a pretrained BERT model, enriching acoustic representations with semantic information.

Chimera (ACL 2021): introduces a shared semantic projection that maps both audio and text into a common space, trained with contrastive loss to align modalities.

Decoder Improvements

COSTT (Conference on Speech Translation, AAAI 2021): a continuous‑generation decoder that first produces an ASR transcript and then the translation, enabling the model to act like a note‑taking interpreter.

3. Training Strategies

Progressive Multi‑Task Learning (XSTNet, InterSpeech 2021)

A unified model jointly learns ASR, MT, and speech translation tasks. Tags (e.g., audio , text ) indicate the modality, allowing the same encoder to process both audio and textual inputs. Pre‑training on large MT corpora followed by multi‑task fine‑tuning yields higher BLEU scores.

Data Augmentation

Pseudo‑labeling (forward translation) generates synthetic speech‑translation pairs from MT data, effectively enlarging the training set. This technique contributed to strong performance in the IWSLT 2021 evaluation.

Results

The proposed system achieved a BLEU score of 31.3 on the 2021 test set, surpassing the baseline (~23) by about eight points and outperforming most competing systems.

Q&A Highlights

Addressing hot‑word intervention in end‑to‑end models may borrow techniques from ASR hot‑word handling and code‑switching.

While end‑to‑end models now slightly outperform cascade systems on benchmarks, cascade remains dominant in production due to data limitations.

The rapid growth of speech translation is driven by the rise of video content, 5G connectivity, and increased compute resources, prompting a shift from text‑centric to multimodal AI.

In summary, the talk covered the motivation, current applications, modeling paradigms, advanced encoder/decoder designs, training tricks, data‑augmentation methods, experimental outcomes, and future directions for speech translation.