How I Won the KDD Cup Using DeepSeek’s Web Interface

Competition Overview

The TAAC × KDD Cup 2026 industrial track required predicting conversion rate (pCVR) on anonymized Tencent advertising data. The dataset contains 35 million samples and 142 features, including dense and sparse user/ad attributes, timestamps, and multi‑domain behavior sequences. The evaluation metric is ROC‑AUC with strict latency limits and no model ensembling.

Engineering Verification: Multi‑GPU Batch Size Inconsistency

Using the provided baseline, Distributed Data Parallel (DDP) read different numbers of Parquet row groups on each GPU, producing uneven batch sizes that triggered recompilation errors with torch.compile. DeepSeek was given the full dataset.py, error logs and desired behavior, and returned a complete replacement.

Solution: introduce a carry buffer that greedily merges incoming batches, outputs only full batches, and discards the final incomplete remainder. A DDP Join step synchronises step counts across GPUs, eliminating deadlocks. This fixed the batch‑size issue and allowed stable speedups from torch.compile.

QueryFormer: Champion Solution

QueryFormer unifies sequence modeling and feature interaction by constructing high‑quality Query tokens from user, ad and dense features, then using those queries to retrieve the most relevant parts of the multi‑domain behavior sequence.

Five‑Step Optimization

Re‑organise non‑sequence features – use the official GroupNSTokenizer to group related user and ad features, projecting each group into an independent token. Dense embeddings are split according to original field boundaries and processed with DCN‑V2 instead of a single MLP, preserving intra‑group structure.

Enrich Query via self‑ and cross‑attention – add QuerySelfAttn (self‑attention within user tokens and within ad tokens) and QueryCrossAttn (cross‑attention between user and ad tokens). Ablation: removing QuerySelfAttn drops validation AUC by 0.00020; removing QueryCrossAttn drops AUC by 0.00029.

Targeted sequence retrieval – replace simple average pooling with QuerySeqCrossAttn, letting the ad‑side query attend to the behavior sequence. Ablation: replacing with average pooling reduces AUC by 0.00023.

Dynamic Query generation – replace the static MLP with SeqQueryCrossAttn, a cross‑attention from the sequence‑derived query back to non‑sequence tokens, producing sample‑dependent queries. This yields the largest single AUC impact (‑0.00038).

Multi‑column embedding matrix – generate H parallel tokenisation pipelines (H=4 in the final configuration) to obtain multiple views of the same input. Scaling experiments show monotonic AUC improvement from H=1 to H=8; H=4 gives validation AUC 0.84606 with acceptable latency.

After all five steps the validation AUC reaches 0.84606 and the test AUC is 0.83254, securing the championship.

DeepSeek Practical Workflow

Research idea → Provide full code, constraints, and expected behavior to DeepSeek → DeepSeek returns complete replaceable code → Run on competition server → Feed back errors, logs, and metrics → DeepSeek revises code

Key practices: provide complete context (full files, error traces, input shapes, environment), request whole files rather than patches, iterate one hypothesis at a time, and let experimental AUC or log‑loss drive decisions.

Training Optimisation

The optimizer used is MuonPlus , which combines dense‑parameter optimisers (Adam‑like) with Adagrad for high‑cardinality sparse embeddings. MuonPlus was previously decisive in the CMI Sensor Data Challenge and NFL Big Data Bowl 2026 (Li et al., 2026).

Additional tricks: CosineAnnealingLR, gradient clipping, EMA, bfloat16 precision, and torch.compile for compilation‑time speedups.

Results

QueryFormer achieved test AUC 0.83254, winning the TAAC × KDD Cup 2026 industrial track.

References

Li, G., Zhang, S., Pan, J., Ning, W., Chen, J., Xue, G., Zhou, C., Huang, S., Gu, H., Yang, M. "Expand More, Shrink Less: Shaping Effective‑Rank Dynamics for Dense Scaling in Recommendation". KDD 2026. https://arxiv.org/abs/2605.23191

Liu, M., Bai, Y., Chan, Z., Chen, S., Sheng, X‑R., Zhu, H., Xu, J., Chen, X. "EST: Towards Efficient Scaling Laws in Click‑Through Rate Prediction via Unified Modeling". arXiv:2602.10811, 2026. https://arxiv.org/abs/2602.10811

Zhou, Y., Ogurtsov, K., Anzelmo, D. 1st Place Solution. CMI – Detect Behavior with Sensor Data, Kaggle 2025. https://www.kaggle.com/competitions/cmi-detect-behavior-with-sensor-data/writeups/cmi-1st-place-solution

Zhou, Y. 2nd Place Solution. NFL Big Data Bowl 2026 – Prediction, Kaggle 2026. https://www.kaggle.com/competitions/nfl-big-data-bowl-2026-prediction/writeups/2nd-place-solution