How Speech Models Turn Waveforms into Computable Tokens

In deep learning, language models operate on discrete symbol sequences (words or sub‑words), while microphones capture continuous waveforms. To enable autoregressive prediction, retrieval, or editing with speech large models, a bridge is required: mapping continuous audio into a sequence of finite symbols (tokens) or structured vectors before feeding them into Transformers, diffusion models, or other sequence models.

Three core contradictions arise:

Continuity : Waveform sampling rates of 16–48 kHz make per‑sample modeling prohibitively expensive.

Abstraction level : Acoustic details (noise, timbre) and semantic content are entangled, demanding hierarchical representations.

Alignment with text : For "listen‑then‑generate" scenarios, audio tokens must align with text tokens in a shared semantic space.

The article categorises the most common industrial and academic pipelines into three overarching routes:

Neural codec + vector quantisation (VQ) producing discrete tokens.

Self‑supervised speech models (SSL) followed by clustering to obtain speech units.

Direct use of continuous embeddings as "soft tokens" without hard discretisation.

Scheme A – Neural Audio Codec + VQ Discrete Tokens

Encoder compresses waveform into frame‑level hidden states, which are then vector‑quantised (VQ, RVQ, FSQ, etc.) into discrete indices forming a token sequence. Decoder reconstructs waveform or mel‑spectrogram from tokens. Representative systems include SoundStream, EnCodec, SpeechTokenizer, and various VQ‑VAE/RVQ models.

Advantages :

End‑to‑end reconstructability: tokens retain acoustic information, suitable for speech generation, editing, and speech LMs.

Controllable frame rate: hop/stride can reduce sequence length to tens or hundreds of tokens per second, easing LM training.

Natural alignment with text LMs: discrete sequence + large codebook mirrors NLP sub‑word tokens.

Drawbacks :

Codebook collapse and quantisation error limit audio quality.

Training complexity: requires adversarial or multi‑resolution reconstruction tricks, high compute and tuning cost.

Semantic‑acoustic entanglement: a single token often encodes both content and timbre, requiring downstream handling for speaker/style.

Typical use cases: speech synthesis, continuation, and any "speech‑large‑model" task where tokens can be directly decoded back to sound.

Scheme B – Self‑Supervised Speech Model + Clustering (Speech Units)

Large‑scale unlabelled audio is used to train SSL models such as wav2vec 2.0, HuBERT, or WavLM, producing continuous frame‑level representations. These representations are clustered (k‑means or other) to obtain discrete units (speech units or pseudo‑phones). HuBERT demonstrates that cluster labels can serve as "teachers" for iterative improvement of representations and units.

Advantages :

No reliance on massive paired text‑audio data; can leverage abundant unlabelled speech.

Units tend to capture linguistic content, aiding unsupervised speech modeling and cross‑language tasks.

Easily combined with ASR, translation, and other mature modules.

Drawbacks :

Cannot directly reconstruct high‑quality waveforms; a separate vocoder is needed.

Cluster count and k‑means initialization are sensitive and affect downstream performance.

Temporal resolution may not align with linguistic units, limiting controllability compared with codec tokens.

Typical use cases: low‑resource speech recognition, speech translation, content modelling, often in conjunction with text tokens for multimodal training.

Scheme C – Continuous Vector "Soft Tokens" (No Hard Discretisation)

Instead of forcing each frame into an integer code, the d‑dimensional vectors output by an SSL or CNN encoder are fed directly to a Transformer (optionally projected to match LLM dimensions) or compressed via Perceiver/Adapter modules.

Advantages :

No quantisation loss; smoother information flow, sometimes better for understanding tasks.

Implementation simplicity: freeze audio encoder and train a projection layer for rapid experimentation.

Drawbacks :

Longer sequences increase memory and attention cost; less compact than discrete tokens.

Does not fit the pure discrete LM story, so generative pipelines may still need a final quantisation step.

Use cases: speech understanding (commands, QA, summarisation) and fast fine‑tuning of multimodal LLMs that align with existing text models.

Scheme D – Text‑Intermediate Representation (ASR → Text Tokens)

Strong ASR first converts speech to a character or phoneme sequence, which is then processed by a text LM. This is technically a cascade, though product terminology may call it "speech‑to‑token".

Advantages :

Directly reuses the most powerful text LMs; if annotation is textual, data collection is straightforward.

Debugging is intuitive because ASR errors are visible.

Drawbacks :

Error propagation; spoken, overlapped, or non‑linguistic sounds (laughter, sighs) are poorly captured.

Latency and real‑time constraints: stitching two models makes true end‑to‑end interaction difficult.

Suitable for dialogue MVPs and assistant‑style products where acoustic style is not critical.

Scheme E – Hierarchical / Multi‑Codebook Tokens (Semantic vs Acoustic)

Multi‑layer RVQ or dual‑branch designs capture semantic/content in one layer and timbre/detail in another. Systems such as AudioLM adopt semantic tokens + acoustic tokens for cascaded generation.

Advantages :

Better decoupling of content and style, facilitating cloning and controllable generation.

Clear division of labour: abstract LM operates on semantic layer, acoustic layer handles fine‑grained rendering.

Drawbacks :

Highest system complexity; long training and inference pipelines with strict alignment and scheduling requirements.

Ideal for high‑fidelity voice cloning, controllable TTS, and research‑grade speech generation models.

Trade‑off Overview

The accompanying matrix summarises common dimensions (discrete vs continuous, reconstruction strength, data dependence, typical challenges). No method is universally optimal; the best choice depends on the target task.

Practical Guidance for Engineers

For pure understanding tasks (classification, commands, summarisation), prefer continuous representations with a projection layer or SSL units + LLM for rapid iteration.

For speech generation or speech LMs, prioritize neural codec discrete tokens or hierarchical semantic/ acoustic tokens.

For quick product rollout, ASR + text LLM often offers the best cost‑performance trade‑off, accepting loss of acoustic nuance.

Evaluation should go beyond objective metrics (WER, mel‑distance); consider MOS, similarity, latency, and streaming capability for generative tasks.

Further Reading

VQ‑VAE / RVQ / SoundStream / EnCodec – discrete acoustic tokens.

wav2vec 2.0, HuBERT, WavLM – SSL representations and speech units.

SpeechTokenizer, Semantic tokens – research and open‑source implementations for content‑acoustic decoupling.