Causal LM vs Prefix LM: Core Differences, Attention Masks, and Choosing the Right Model

Definition of Causal Language Model (Causal LM)

A Causal LM (also called an autoregressive model) predicts the next token using only tokens that appear to its left. During generation the model sees the already generated sequence but never the future tokens, which enforces a strict left‑to‑right processing order. The GPT series is a canonical example of this architecture.

Definition of Prefix Language Model (Prefix LM)

A Prefix LM splits the input into two segments:

Prefix : a block of tokens that the model can attend to bidirectionally (full self‑attention). This segment provides complete contextual understanding.

Generation part : tokens that must be produced autoregressively. When generating token D, the model can attend to the entire prefix and to D itself; token E can attend to the prefix plus D, but not to any later token.

Models such as GLM and UniLM adopt the Prefix LM architecture, effectively combining an encoder‑like read phase with a decoder‑like write phase while sharing parameters.

Attention‑Mask Design

Causal LM mask is a lower‑triangular matrix where each token can attend only to itself and earlier tokens:

Token A: attends to A

Token B: attends to A, B

Token C: attends to A, B, C

Token D: attends to A, B, C, D

Token E: attends to A, B, C, D, E

Prefix LM mask contains two distinct regions:

Within the prefix (e.g., A B C) all tokens attend to each other (full bidirectional attention).

In the generation segment the mask reverts to causal behavior: D attends to A, B, C, D; E attends to A, B, C, D, E.

This hybrid mask enables full visibility inside the prefix and strict left‑to‑right visibility for the generated tokens.

Design Rationale and Trade‑offs

Causal LM uses a single, uniform training objective—predict the next token given all previous tokens. The simplicity yields high training efficiency and scalability, which is why it is preferred for very large pre‑training runs (e.g., GPT‑3, GPT‑4).

Prefix LM behaves like a lightweight encoder‑decoder: the prefix acts as an encoder that builds a rich representation of the context, while the generation part acts as a decoder that produces output conditioned on that representation. This shared‑parameter design provides greater flexibility, allowing the same model to be fine‑tuned for both understanding tasks (classification, QA) and generation tasks (text completion, summarisation) without needing separate architectures.

Practical Guidance for Model Selection

Choose a Causal LM when the primary requirement is fast, pure generation at massive scale, or when training efficiency is the dominant concern.

Choose a Prefix LM when you need a single model that can both comprehend a given context and generate continuations, such as in multi‑task learning, instruction‑following, or fine‑tuning scenarios that involve both classification and generation.

In summary, Causal LM processes tokens strictly forward and is optimized for generation, whereas Prefix LM first reads the full prompt bidirectionally and then writes, offering combined comprehension and generation capabilities.