Essential Transformer Interview Cheat Sheet: 11 Must‑Know Q&A

Interviewers often expect concise, technically accurate answers about Transformers, and many candidates rely on a set of recurring "canned" responses. While these observations stem from experimental practice rather than formal theory, mastering them can significantly improve interview performance.

Key Principles Behind the Canned Answers

The recurring themes focus on improving data distribution, facilitating model training, and enhancing expressive power. By ensuring inputs to softmax are well‑scaled, keeping gradients in a sensitive range, and using normalization layers, models become more stable and easier to train.

Interview Q&A

Write the self‑attention formula. Softmax(QK / sqrt(d_k)) V Why scale the QK product? Scaling improves the softmax input distribution, keeping values in a gradient‑sensitive range, which prevents vanishing gradients and makes the model easier to train.

Must self‑attention be expressed exactly this way? No. Any mechanism that captures similarity or correlation can be used, provided it is fast, easy to learn, and sufficiently expressive.

Are there alternatives that avoid dividing by √dₖ? Yes. Any method that keeps per‑layer gradients within a sensitive range works, such as careful initialization (e.g., the approach used in Google’s T5 model).

Why does a Transformer use Layer Normalization? LayerNorm improves the distribution of layer inputs, keeping values in a gradient‑sensitive range and preventing vanishing gradients, thereby facilitating training.

Why not use Batch Normalization? NLP sequences have variable lengths and padding tokens, which distort batch statistics; additionally, large Transformer models often use small batch sizes, leading to instability.

Why does BERT add a positional embedding? Because the vanilla Transformer is position‑agnostic; positional embeddings inject location information, enhancing the model’s expressive capacity.

Why can BERT’s three embeddings be summed? The token, segment, and positional embeddings reside in the same vector space, allowing a simple linear combination without harming performance.

Why does a Transformer use three distinct matrices Q, K, V? Using separate Q, K, and V matrices increases the model’s capacity and expressive power.

Why employ multi‑head attention? Multiple heads further expand capacity and expressive ability by allowing the model to attend to information from different representation subspaces.

Why does BERT use BPE sub‑word tokenization? BPE efficiently handles out‑of‑vocabulary words and provides a granularity that balances semantic richness with manageable vocabulary size.

When discussing these points in an interview, providing brief theoretical motivation (e.g., gradient scaling, normalization benefits) alongside practical examples demonstrates a solid grasp of Transformer fundamentals.