How Audio Waveforms Are Turned Into Model‑Readable Tokens

Large text models ingest discrete characters or sub‑words, but a microphone records a continuous waveform that looks like a line on an ECG trace. Feeding every sample point (every few tenths of a millisecond) would produce an astronomical sequence length, making attention and memory costs infeasible.

Two engineering steps

Compress time : merge fine‑grained samples into coarser "beats", e.g., a feature frame every few milliseconds.

Compress information : encode each frame either as a discrete ID (like a page number in a dictionary) or as a vector (a coordinate point). Without these steps, pre‑training and alignment would be like building a sandcastle on a beach.

Three common “cross‑river” approaches

Approach A – Neural audio codec (lossy compression)

Encoder compresses the waveform into a latent representation, then vector‑quantizes it into a limited set of discrete codes; the decoder tries to reconstruct the sound. The intuition is similar to JPEG: some acoustic detail is discarded, yet the human ear can still recognize speech. This method suits voice synthesis, cloning, and continuation because the token still carries information about "how to sound". Drawbacks include potential "machine‑like" artifacts if the codebook is poorly designed, and audio quality is bounded by the codec’s capacity.

Approach B – Self‑supervised representation + clustering

The model first learns on massive unlabeled speech to discover acoustic frames that look similar, then clusters them into "speech units". This works when high‑quality textual labels are missing or in multilingual scenarios. The downside is that these units are not optimized for high‑fidelity reconstruction; high‑quality synthesis often still requires a separate vocoder or additional modules.

Approach C – Continuous vectors as "soft tokens"

Each frame or chunk is represented by a high‑dimensional vector fed directly to downstream networks. This is ideal for tasks that only need to "understand" speech, such as spoken QA, command following, or content comprehension, because the pipeline is relatively straightforward. The trade‑off is longer sequences, higher memory and inference cost, and a slight performance gap compared to pure discrete language‑model pipelines.

Mixed‑strategy systems

Real products often combine approaches: upstream may use continuous vectors for robustness and downstream quantizes them for generation; or employ multi‑layer tokens—one layer focusing on "what to say" and another on "how to say it", akin to separating plot and dialogue.

Common pitfalls for newcomers

Pitfall 1 : Assuming that having an ASR system eliminates the need for tokenization. ASR still maps audio to textual symbols (characters or phonemes).

Pitfall 2 : Believing discrete tokens are always better. Discretization loses information; continuous vectors can be smoother for understanding‑oriented tasks.

Pitfall 3 : Handing the entire "human‑like speaking" problem to the text side. Text‑only models may produce fluent reading but can sound unnatural; speech synthesis and interaction timing have separate considerations.

Bottom line: audio must first be transformed into a sequence of tokens or vectors that the model can handle.