How to Ace Multimodal Model Interviews at Taobao's Search AI Division

First Round

Self‑introduction and project overview, with discussion of methods used and feasibility.

Introduction to CLIP.

Understanding of LoRA and its fine‑tuning principle.

Familiarity with various multimodal large models; brief introductions.

BLIP's three loss functions and data cleaning process.

Improvements of BLIP2 over BLIP and further enhancements in BLIP3.

Three training procedures of Qwen‑VL and their purposes.

Comparison of adaptor designs when connecting visual encoders to LLMs: Q‑Former (complex) vs. simple MLP (as in LLaVA), discussing pros and cons.

Code task: implement multi‑head self‑attention.

The first interview is fairly standard; knowing common multimodal models and the design motivations behind them is sufficient.

Second Round

Self‑introduction and project recap, probing the motivation behind chosen methods and potential issues.

Understanding of the Transformer architecture, differences between encoder and decoder attention, and the reason for scaling by \(\sqrt{d_k}\) in attention computation.

Discussion of classic Transformer‑based language models, structural changes in Qwen compared to the original Transformer, and further improvements in Qwen2.

Knowledge of RLHF, differences between DPO and PPO, loss formulations, and their respective strengths and weaknesses.

Review of CLIP and other contrastive learning methods.

Open‑ended question about known multimodal large models and the biggest challenges they face.

Code task: solve the Longest Common Subsequence problem (LeetCode 1143).

The second interview remains conventional but probes deeper understanding of model internals and breadth of knowledge, making it slightly harder than the first.

Third Round

Self‑introduction followed by an in‑depth project discussion.

Discussion of major large‑model and multimodal‑model families (Transformer, BERT, GPT, LLaMA, Qwen, etc.) and the evolution of models up to the latest inference‑only models like o1.

Experience with training large models, even on a smaller scale.

Casual conversation covering career planning and other topics.

The third interview is relaxed; the interviewer reviews previously covered topics and focuses on overall understanding and fit, typically lasting around 40 minutes.

Overall Summary

The interview experience was positive, with approachable questions and supportive interviewers. The first two rounds followed a predictable pattern of technical and conceptual questions, while the third round was more conversational, reflecting a senior‑level assessment of the candidate's grasp of multimodal large‑model concepts.