30+ Expert Q&A on Large Language Model Architecture, Training, and Deployment

This collection presents a series of interview questions and concise expert answers about large language models (LLMs). It covers model architectures, attention mechanisms, context handling, tokenization, embedding techniques, training strategies, inference optimizations, retrieval‑augmented generation, prompt engineering, safety, fine‑tuning, and model distillation. The material is organized by question number, with occasional diagrams illustrating concepts.

Q1: What are the advantages and disadvantages of encoder‑only (BERT‑style), decoder‑only (GPT‑style), and full encoder‑decoder architectures?

Answer not provided in the source.

Q2: How does the self‑attention mechanism enable LLMs to capture long‑range dependencies, and how does it differ from RNNs?

Self‑attention allows every token to interact directly with all other tokens by computing similarity between Query and Key vectors, producing a weighted aggregation of global information in a single step. This eliminates the need for sequential information propagation, avoiding gradient vanishing and enabling parallel computation, which makes training much faster than the step‑by‑step processing of RNNs.

Q3: Why do LLMs have a notion of context length, and why does it refer to the total length of input and output tokens?

Transformer models encode input tokens into a fixed‑length sequence and use positional encodings to preserve order. Because attention is quadratic (O(n²)) in token count, memory and compute costs rise sharply with longer sequences, so a maximum context length (e.g., 2048 or 4096 tokens) is set during training. The context length counts both the tokens fed to the model and the tokens the model has already generated, because autoregressive models must re‑read the entire visible context for each new token.

Q4: How does a large‑model tokenizer differ from traditional Chinese word segmentation, and is there a unique tokenization for a given sentence?

LLM tokenizers are designed to produce model‑compatible token IDs, while traditional Chinese segmentation aims at human‑readable word boundaries. For a fixed vocabulary, the tokenizer follows a longest‑match greedy algorithm, producing a deterministic token sequence for a given sentence.

Q5: How does a large model distinguish user utterances from AI responses in a chat history?

The model relies on explicit role markers such as <|im_start|>user and <|im_start|>assistant. These markers do not convey true understanding; the model has simply learned the pattern that follows each marker.

Q6: What are the differences between static word embeddings (e.g., word2vec) and context‑aware embeddings from large models, and does the latter render static embeddings obsolete?

Static embeddings are lightweight, fast, and easy to train on massive corpora; they offer clear interpretability and are useful when resources are limited.

Contextual embeddings capture token meaning in the surrounding context, providing richer semantics for downstream tasks such as classification.

However, contextual embeddings are more computationally expensive and may not always preserve simple linear relationships like king – man + woman ≈ queen.

Q7: Why does the word2vec vector space exhibit linear relationships such as king – man + woman ≈ queen , and do LLM token embeddings show similar properties?

Word2vec learns vectors that encode both similarity and certain relational directions. LLM embedding layers also retain some linear relationships, especially in early layers, but deeper transformer layers produce highly contextualized vectors where simple linear analogies may no longer hold.

Q8: How does the attention mechanism compute relevance between tokens, and does each attention head focus on a single token?

Attention computes three steps: (1) similarity scores via Query‑Key dot products, (2) softmax normalization to obtain weights, and (3) weighted sum of Value vectors. Each head attends to the whole sequence, learning diverse patterns (e.g., syntax, keyword matching). Multi‑head attention concatenates these diverse views.

Q9: To make a model forget a specific piece of knowledge, should we modify the attention layer or the feed‑forward network?

Modifying the attention layer (especially the Q/K matrices) is more efficient because attention directly controls token‑to‑token interactions; changing a small fraction of parameters can effectively cut the undesired knowledge pathways.

Q10: Why does FlashAttention improve inference speed despite not reducing the total number of arithmetic operations?

FlashAttention optimizes memory access by processing attention in blocks, using a streaming softmax that updates intermediate results on the fly, and reducing data movement. This lowers latency and improves GPU utilization, yielding faster inference.

Q11: What are the advantages of RoPE (Rotary Positional Embedding) over absolute positional encodings, and what challenges arise when extrapolating to longer contexts?

RoPE encodes relative positions, making it easier for the model to learn distance relationships.

It can be naturally extended to longer sequences because the rotation angle formula can continue beyond the training length.

When positions become very large, the rotation angles may become unstable, causing the model to lose accurate long‑range information.

Q12: When should one prefer zero‑shot classification over a classifier built on embeddings and a logistic regression layer?

When labeled data are scarce or noisy.

When label sets change frequently, avoiding costly re‑training.

When rapid deployment is needed without a full training pipeline.

Q13: How does BERT’s masked language modeling differ from GPT’s next‑token prediction, and why does this help downstream text‑classification tasks?

BERT uses masked tokens ( [MASK]) and learns to predict them, forcing the model to understand the whole sentence. This bidirectional pre‑training yields richer representations that transfer well to classification tasks.

Q14: In a step‑by‑step classification pipeline with 1 000 labeled samples and 1 000 000 unlabeled comments, how should one combine a representation model and a generative model?

Fine‑tune a BERT‑style encoder on the 1 000 labeled examples.

Generate pseudo‑labels for high‑confidence unlabeled samples and add them to the training set.

Use a large generative LLM (via few‑shot prompting) to assist the encoder, merging predictions for improved accuracy.

Q15: After adding a powerful generative model, what is the remaining role of embedding models?

Embedding models remain valuable for fast, low‑cost similarity search, clustering, and as lightweight components in pipelines where full generation is unnecessary.

Q16: How do bag‑of‑words and document‑embedding approaches differ, and is a bag‑of‑words model still useful?

Bag‑of‑words is fast, low‑resource, and highly interpretable.

Document embeddings capture semantic similarity and support downstream neural methods.

Q17: How does c‑TF‑IDF differ from traditional TF‑IDF, and how does it improve topic quality?

c‑TF‑IDF computes term frequencies and inverse document frequencies at the class (topic) level, emphasizing words that are distinctive for a specific topic while down‑weighting ubiquitous terms, leading to clearer, more discriminative topic keywords.

Q18: What are the pros and cons of centroid‑based (e.g., K‑means) versus density‑based (e.g., DBSCAN) text clustering?

Centroid‑based: simple, fast, scalable, but requires pre‑defining the number of clusters and struggles with non‑convex shapes.

Density‑based: discovers arbitrarily shaped clusters and handles noise, but is sensitive to hyper‑parameters and less scalable.

Q19: How can one improve topic separation when many keywords overlap across topics?

Apply c‑TF‑IDF to down‑weight shared high‑frequency words.

Adjust the number of topics and use regularization to increase inter‑topic distance.

Employ density‑based clustering (e.g., DBSCAN) instead of K‑means.

Post‑process keywords by filtering or re‑weighting overlapping terms.

Q20: How should temperature and top‑p be set for translation, creative writing, and brainstorming tasks?

Use low temperature for precise tasks like translation; increase temperature and top‑p for creative or brainstorming tasks to encourage diversity.

Q21: What are the essential components of a professional prompt template, and why is role definition important?

Role definition (who the model is).

Task description.

Context information.

Input data.

Output format requirements.

Explicit role definition guides the model to adopt the appropriate tone and behavior.

Q22: How can one design prompts to prevent prompt‑injection attacks, and how can the system detect such attacks?

Use fixed role statements (e.g., "You are a finance‑only assistant").

Adopt a structured template like "Instruction: …\nInput: …\nOutput:".

Restrict user‑provided content to a designated slot.

Implement keyword blacklists, semantic analysis, and response monitoring to catch violations.

Q23: How to make a model “think” before answering when it lacks a dedicated reasoning module?

Include explicit instructions such as "Think step‑by‑step before answering" (chain‑of‑thought).

Ask the model to verify consistency (self‑consistency).

Prompt for multiple possible scenarios (tree‑of‑thought).

Q24: When a document exceeds the model’s context window, how can one combine summarization and RAG to answer both high‑level and detailed questions?

Generate a global structured summary for overview queries.

Chunk the document, index each chunk, and retrieve relevant pieces for detailed queries.

Merge summary and retrieved chunks as context for the final answer.

Q25: Why does CLIP maximize similarity for matching image‑text pairs while minimizing it for mismatched pairs?

Maximizing alignment forces image and text embeddings into a shared semantic space; minimizing mismatched pairs prevents the model from learning trivial modality‑specific features, improving cross‑modal retrieval.

Q26: Why does BLIP‑2 insert a Q‑Former between the visual encoder and the language model?

Bridges modality gaps by compressing dense visual patches into a manageable set of learnable queries.

Reduces input length and computational load.

Enables cross‑attention that selects the most relevant visual information for language generation.

Q27: How can a weak multimodal model and a strong text model be combined to answer multimodal questions?

Use the weak multimodal model to generate a caption, object list, or structured visual summary.

Feed this intermediate representation together with the original question into the strong text model for final reasoning and answer.

Q28: How to build an AI photo assistant that indexes millions of images and retrieves relevant photos efficiently?

Pre‑process photos (deduplication, quality check, format standardization).

Extract visual features with a pretrained encoder (e.g., CLIP, ResNet).

Store features in a vector database (FAISS, Milvus) for fast similarity search.

Convert user queries to text embeddings and perform nearest‑neighbor retrieval.

Rank and filter results using metadata (time, location, tags) and allow user feedback for continual improvement.

Q29: Why are dual‑encoder architectures preferred over cross‑encoders for large‑scale similarity search?

Dual encoders allow offline pre‑computation of all item vectors, enabling sub‑second retrieval with vector indexes.

Cross‑encoders require joint encoding of query and candidate each time, which is computationally prohibitive at scale.

Q30: What are the pros and cons of MNR (multiple negative ranking) loss, cosine similarity loss, and softmax loss for embedding training, and when might cosine loss be preferable?

MNR: strong discriminative power with many negatives; higher computational cost and sensitive to negative sampling.

Cosine loss: simple, directly optimizes angular similarity; may be less effective when many hard negatives are needed.

Softmax loss: stable training as a classification problem; limited negative diversity.

Cosine loss is suitable when training resources are limited, the dataset is small, or when embeddings must remain normalized.

Q31: How to generate hard negatives to improve model performance?

Random negative sampling (baseline).

Hard negative mining: select negatives with high similarity to positives.

Dynamic hard negative mining during training.

Use a current model to retrieve top‑confusing negatives, or apply adversarial generation.

Q32: Why does TSDAE use a special token (e.g., [CLS] ) instead of mean‑pooling for sentence representation?

The special token is explicitly trained to aggregate whole‑sentence information, yielding richer semantics.

It avoids noise from padding or irrelevant tokens, and is computationally cheaper than averaging all token vectors.

Q33: How does MTEB improve upon STSB, and which embedding tasks does it cover?

MTEB expands the benchmark suite to many more datasets, covering semantic similarity, retrieval, classification, clustering, ranking, and other multimodal or cross‑lingual tasks, providing a unified evaluation framework.

Q34: When training data are scarce, how can SetFit enlarge the effective dataset?

Encode the few labeled examples with a lightweight sentence‑embedding model.

Cluster or retrieve unlabeled sentences that are close to the labeled ones.

Assign pseudo‑labels to these nearest neighbors and fine‑tune a simple classifier on the enlarged set.

Q35: How to continue pre‑training a model on domain data while preserving its general capabilities?

Mix domain and general corpora during continued pre‑training.

Use a small learning rate and regularization (e.g., L2 or knowledge distillation) to limit drift.

Apply multi‑task objectives that retain generic skills.

Optionally freeze lower layers and only fine‑tune higher layers.

Q36: In medical text classification, compare (a) fine‑tuning a generic BERT, (b) further pre‑training BERT on medical text then fine‑tuning, and (c) training a model from scratch on medical data.

(a) Low cost, quick, but limited domain performance.

(b) Balances domain adaptation and retained general knowledge; moderate cost.

(c) Highest potential performance but very high computational cost and data requirements.

Overall, (b) is the most common trade‑off in practice.

Q37: How to handle token‑level label alignment for NER when BERT splits a word into sub‑tokens?

Label only the first sub‑token and ignore the rest (or assign a special "X" label).

Copy the original word label to all sub‑tokens.

Use a sub‑word‑level labeling scheme (more complex and rarely needed).

Q38: How to improve a primarily English embedding model’s Chinese performance with low‑cost continued pre‑training?

Continue pre‑training on a modest Chinese corpus.

Mix English and Chinese data to avoid catastrophic forgetting.

Expand the tokenizer with Chinese characters.

Apply cross‑lingual alignment tasks (e.g., translation pairs) to bridge the language gap.

Q39: How to verify a claim that a text was generated by DeepSeek‑R1 using the provided prompt?

Run DeepSeek‑R1 with the same prompt and compute perplexity on the target text.

Low perplexity indicates the model likely generated it; high perplexity refutes the claim.

Q40: How to fine‑tune a Llama model to produce concise, WeChat‑style dialogue while meeting domestic safety standards?

Collect a large corpus of real WeChat conversations.

Filter out unsafe or prohibited content.

Fine‑tune with a low learning rate, emphasizing brevity and colloquial tone.

Incorporate safety alignment via RLHF or rule‑based post‑processing.

Q41: What is the benefit of block‑wise quantization in QLoRA compared with naïve quantization?

Each block gets its own scaling factor, preserving local distribution details.

Reduces quantization error accumulation across the whole weight matrix.

Maintains high compression while keeping model accuracy.

Q42: How to convert a corporate knowledge base into an SFT dataset?

Split long articles into manageable paragraphs or QA pairs.

Structure each example as a {"prompt": ..., "response": ...} JSONL entry.

Clean the data, remove duplicates, and optionally generate multiple question variants for each answer.

Q43: Compare PPO and DPO – their advantages and drawbacks.

PPO: Direct reinforcement learning, can optimize arbitrary behavior, but training is complex and requires a reward signal.

DPO: Learns from preference data without environment interaction, simpler and faster, but depends on high‑quality human preferences.

Q44: In PPO fine‑tuning, how to avoid loss of generalization and prevent the model from collapsing to a single high‑reward answer?

Mix in original pre‑training data or diverse samples during RL.

Apply KL‑divergence regularization to keep the new policy close to the base model.

Use entropy regularization or reward shaping to encourage answer diversity.

Q45: How to turn average user dwell time on AI‑generated articles into DPO preference data for platforms like Xiaohongshu and Zhihu?

Treat longer dwell time as a higher preference signal.

Form pairs of articles where one has significantly higher dwell time and label the pair as "prefer A over B".

For Xiaohongshu, combine dwell time with multimodal signals (images, likes). For Zhihu, dwell time correlates more directly with content quality, so simple time‑based pairs suffice.

Q46: When should each technique (Prompt Engineering, RAG, SFT, RL, RLHF) be applied?

Prompt Engineering – quick prototyping, lightweight personalization, no training cost.

RAG – augmenting factual knowledge, accessing external databases, improving answer accuracy.

SFT – shaping output format, language style, and domain‑specific behavior.

RL – teaching complex reasoning, tool use, multi‑step planning.

RLHF – continuously improving model quality based on human feedback, handling safety and preference.

Q47: How does DeepSeek‑R1 differ from DeepSeek‑R1‑Zero, and why is the latter still useful?

R1‑Zero is trained only with zero‑shot data; it is cheap and serves as a baseline or a teacher for distillation.

R1 adds supervised fine‑tuning and RLHF, yielding readable reasoning and higher accuracy.

R1‑Zero’s value lies in rapid prototyping, providing initial weights for later distillation, and serving scenarios where interpretability is not critical.

Q48: How does DeepSeek distill R1’s reasoning ability into a smaller model, and how to perform a similar vertical‑domain distillation?

Generate step‑by‑step reasoning examples with R1 (teacher).

Supervise the smaller model to mimic both the final answer and the intermediate reasoning steps.

Use a dedicated “process consistency” loss to align the student’s reasoning trajectory with the teacher’s.

For a vertical domain, collect domain‑specific prompts, let R1 produce detailed solutions, and fine‑tune the student on this curated dataset.

Q49: How to extend R1‑Zero’s zero‑shot approach to subjective tasks such as creative writing or strategic analysis?

Introduce diverse evaluation metrics (creativity, coherence, relevance) and incorporate them into a reward model.

Combine zero‑shot generation with human‑in‑the‑loop feedback (RLHF) to guide the model toward desirable subjective qualities.

Enrich the zero‑shot dataset with varied examples to increase stylistic diversity.

Q50: What resources are needed to train a model that solves integer arithmetic (0‑1000) with <1% error using RL?

Start from a base model of at least 7 B parameters to have sufficient reasoning capacity.

Hardware: 2 × NVIDIA A100 80 GB (or equivalent) for environment simulation and policy updates; scaling to 4 × A100 can accelerate training.

Training time: roughly 12–24 hours for supervised warm‑up, plus 24–48 hours of PPO fine‑tuning, depending on batch size and reward design.

Q51: What are the hardware and time estimates for RL‑fine‑tuning QwQ‑32B for a specialized research assistant?

Dataset: domain papers, code snippets, expert‑annotated Q&A.

Hardware: 8 × A100 80 GB (or 8 × H100) – 4 GPUs for environment simulation, 2 for policy inference, 2 for gradient updates.

Training schedule: ~20 h of supervised fine‑tuning, followed by 48 h of PPO on math/code tasks and 24 h of RLHF on research‑style dialogues. Total ≈ 4 days.

Overall, the article provides a thorough technical reference for large‑model interview preparation, covering theory, practical engineering tricks, and advanced training methodologies.