Crack Large-Model Interviews: Master Positional Encoding, Residuals, LayerNorm & FFN

Positional Encoding in Transformers

Standard self‑attention is permutation‑invariant, meaning it cannot distinguish the order of tokens. Positional encoding injects order information so that the model can differentiate "I love Beijing" from "Beijing loves me".

Absolute sinusoidal encoding

Uses fixed sine and cosine functions of different frequencies for each position: PE_{(pos,2i)} = sin(pos/10000^{2i/d}), PE_{(pos,2i+1)} = cos(pos/10000^{2i/d}).

Advantage: the same functions can be evaluated for positions longer than those seen during training, enabling extrapolation.

Disadvantage: limited expressive power because the pattern is predetermined.

Learnable positional encoding

Each position index has a dedicated trainable vector that is added to the token embedding.

Advantage: the model can adapt the encoding to the data, often yielding higher accuracy on the training length.

Disadvantage: poor extrapolation; a model trained on a maximum length of 512 tokens cannot reliably process longer sequences.

Relative positional encoding

Encodes the distance between two tokens rather than their absolute indices. The attention score is modified as score_{ij} = (Q_i K_j^T) + a_{i-j}, where a_{i-j} is a learned bias for relative offset i-j.

Advantage: aligns with the intuition that language depends on relative order and works better on long sequences.

Rotary Positional Embedding (RoPE)

Applies a rotation to the query and key vectors based on token position: Q_i' = R_{pos_i} Q_i, K_i' = R_{pos_i} K_i, where R_{pos} is a block‑diagonal rotation matrix derived from sinusoidal frequencies.

Combines absolute position (through the rotation) with an implicit relative‑distance term, because the inner product of rotated vectors yields a cosine of the relative offset.

Benefits: elegant theoretical justification, strong extrapolation to longer texts, and low overhead. Adopted by LLaMA, Qwen and many recent large language models.

Residual Connections and Normalization

Both components are essential for training deep Transformer stacks.

Residual (skip) connections

Form: output = LayerNorm(x + Sublayer(x)).

They provide a direct gradient path, mitigating vanishing gradients and allowing deeper networks to converge.

Layer Normalization (LayerNorm)

Normalizes each token’s hidden vector across the feature dimension: LN(x) = (x - μ)/σ * γ + β, where μ and σ are computed per token.

Why not BatchNorm? BatchNorm depends on batch statistics, which vary with batch size and sequence length, making it unstable for variable‑length NLP data. LayerNorm is independent of batch size.

RMSNorm (Root‑Mean‑Square Normalization)

Variant of LayerNorm that removes the mean subtraction: RMSNorm(x) = x / RMS(x) * γ + β, where RMS(x) = sqrt(mean(x^2)).

Advantages: fewer arithmetic operations, lower memory bandwidth, and comparable performance. Used in recent LLaMA and Qwen models.

Feed‑Forward Network (FFN) in a Transformer Block

The FFN adds non‑linear transformation capacity after the attention sub‑layer.

def transformer_block(x):
    # Multi‑head self‑attention
    attn = MultiHeadSelfAttention(x)
    x = LayerNorm(x + attn)
    # Feed‑forward network
    ffn = Linear(ReLU(Linear(x)))   # classic: ReLU activation
    # Alternative activations: GeLU, SwiGLU (x * SiLU(x))
    x = LayerNorm(x + ffn)
    return x

Typical architecture: two linear projections with a hidden dimension usually 4× the model dimension, separated by an activation.

Modern variants replace ReLU/GeLU with SwiGLU , which computes x * SiLU(x) and often yields better performance for large models.

The FFN operates independently on each token, enriching the representation before the next attention layer.

Key Takeaways for Model‑Level Understanding

Self‑attention is the core mechanism, but positional encoding, residual connections with normalization, and the FFN are the “blood‑vessel” components that make deep Transformers trainable and expressive.

When discussing positional encoding, mention both absolute (sinusoidal) and relative schemes, and be prepared to explain why RoPE is currently favored in large‑scale LLMs.

Explain that LayerNorm is chosen over BatchNorm for its independence from batch statistics, and note RMSNorm as a lightweight alternative.

Describe the FFN as the module that injects non‑linearity and expands model capacity, and cite common activation choices (ReLU, GeLU, SwiGLU).