10 Must‑Know Tencent AI Interview Topics: Overfitting, Dropout, Transformers & Beyond

A few weeks ago a colleague interviewed for a Tencent algorithm position and shared the core Q&A details as a reference for future candidates.

The interview focused on ten technical topics.

1. Why does overfitting happen? How to mitigate it?

Overfitting occurs when a model learns the training data too well, including noise, leading to poor performance on new data. It can be explained by the bias‑variance decomposition: high variance (large model complexity) causes the model to memorize noise.

Bias : systematic error due to insufficient model capacity.

Variance : sensitivity to training‑set fluctuations.

Irreducible error : inherent data noise.

Mitigation strategies are organized into three layers:

Data layer – increase data size, clean noisy samples, remove outliers.

Model layer – reduce complexity, prune parameters, apply feature selection or dimensionality reduction (e.g., PCA, L1 regularization).

Training layer – use regularization (L1/L2, Dropout), early stopping, cross‑validation, and ensemble methods.

When answering, structure the response as “Data → Model → Training” and provide concrete examples.

2. Explain regularization forms, gradient derivation, and how it reduces overfitting.

Regularization adds a penalty term to the loss to constrain parameter magnitude.

L2 (Ridge) : adds λ‖w‖₂² to the loss, shrinking weights toward zero while keeping them non‑zero, which corresponds to a Gaussian prior.

L1 (Lasso) : adds λ‖w‖₁, encouraging sparsity by driving some weights exactly to zero, equivalent to a Laplace prior.

Elastic Net combines L1 and L2 to balance sparsity and stability.

Regularization reduces overfitting by lowering model variance at the cost of a slight bias increase, improving overall generalization error.

3. How is generalization error generated? Reduce it from bias–variance–noise perspective.

Generalization error is the error on unseen data. It can be decomposed into bias, variance, and irreducible noise.

Bias : error from overly simple models.

Variance : error from overly complex models that fit noise.

Noise : unavoidable data randomness.

To reduce it:

Decrease bias by using more expressive models or richer features.

Decrease variance by adding data, applying regularization, early stopping, or ensemble methods.

Mitigate noise through data cleaning and augmentation.

4. What is the principle of Dropout? Differences between training and testing phases.

Dropout randomly masks a subset of neurons during training, forcing the network to rely on many different paths and preventing co‑adaptation.

During training a Bernoulli mask m is applied to the layer output h and the result is scaled by 1/p (inverted dropout) to keep the expected activation unchanged.

Dropout’s core idea is to randomly silence neurons during training so the model does not over‑depend on specific features, effectively training an ensemble of sub‑networks.

In the testing phase the mask is removed and no scaling is needed because the training scaling already compensates.

5. What is the role of residual connections in Transformers?

Residual connections address gradient vanishing in deep networks and allow the model to learn identity mappings when needed.

Each layer computes y = LayerNorm(x + F(x)), where F(x) is the sub‑layer (e.g., multi‑head attention or feed‑forward). If F(x) learns to output zero, the block behaves as an identity function, preserving information.

6. What do Attention, Normalization, and Embedding mean in Transformer models?

Attention lets the model weigh the relevance of other tokens when processing a token. The common form is Scaled Dot‑Product Attention: Attention(Q,K,V)=softmax(QKᵀ/√d_k)V.

Normalization (LayerNorm) standardizes activations per sample, improving training stability and avoiding gradient explosion. Unlike BatchNorm, it does not depend on batch statistics.

Embedding maps discrete tokens to continuous vectors. Position encoding is added to inject order information because self‑attention is permutation‑invariant.

7. What improvements does BART have over BERT?

BART uses an encoder‑decoder (seq2seq) architecture, enabling both understanding and generation tasks, whereas BERT is a pure encoder for classification‑type tasks.

Its pre‑training is a denoising auto‑encoder: the input is corrupted (deletion, permutation, span masking) and the model learns to reconstruct the original text, providing richer generation capability.

8. How to build your own Instruction‑Tuning dataset?

Instruction‑tuning data consist of (instruction, input, output) triples.

Steps:

Define the target task and format.

Collect raw data from public sources, internal logs, or synthetic generation.

Clean and de‑duplicate data, then format each example as:

Instruction: <instruction>
Input: <input>
Output: <output>

Augment with synonym rewrites, input variations, or controlled noise while keeping the output consistent.

Split into training (≈80%), validation (≈10%), and test (≈10%) sets.

Emphasize data quality and diversity to avoid the model memorizing fixed patterns.

9. Methods to solve LLM hallucination.

Approaches are grouped into data, model, and inference layers.

Data : high‑quality annotations, knowledge‑graph integration, retrieval‑augmented generation (RAG).

Model : post‑processing confidence calibration, adversarial training, temperature tuning.

Inference : Chain‑of‑Thought prompting, factual verification, constrained decoding (beam search, nucleus sampling).

Combining RAG with CoT is a practical way to reduce hallucinations.

10. Why does GAN suffer more from mode collapse than Diffusion models?

GANs train via a min‑max game where the generator tries to fool a discriminator. If a subset of samples consistently deceives the discriminator, the generator may focus on those modes, ignoring others (mode collapse).

Diffusion models learn the full data distribution by progressively denoising Gaussian noise. The multi‑step denoising process provides a global distribution constraint, making the training more stable and less prone to collapse.

Overall, Tencent’s interview emphasizes deep technical understanding, the ability to explain “what” and “why”, and practical experience with modern AI models.