QueryFormer: Winning KDD Cup 2026 with Cross-Attention Query Generation

Introduction

The Tencent Advertising Algorithm Competition (TAAC) 2026, in collaboration with KDD Cup, launched the Tencent UniRec Challenge focusing on unified modeling of sequential behavior and feature interaction for large-scale recommendation. The industrial track targets post-click conversion rate (pCVR) prediction: given user, candidate ad, and context, predict the probability of conversion after click. Models must handle non-sequential multi-field features and multi-domain historical behavior sequences under inference latency constraints.

This article presents QueryFormer , the winning solution by team "日之光面" (Wuhan University and Sun Yat-sen University). QueryFormer connects feature interaction and behavior modeling via a cross-attention-driven query generation mechanism. The paper (arXiv:2609.16548v1) describes architecture design, ablation experiments, and multi-dimensional scaling analysis. Official test AUC: 0.83254 .

Motivation: Query Generation in Unified Modeling

Recommendation models must fuse two information types: multi-field features (user, item, context) and behavior sequences (user history across domains/time). Token-based unified architectures map heterogeneous inputs to a shared representation space. The Query is the key representation linking non-sequential features to behavior sequences: it computes attention weights over sequence tokens, determining which historical information is aggregated.

Existing methods often generate Query by concatenating non-sequential features and pooled behavior representations, then applying an MLP: Query = MLP(N ⊕ Pool(B)). This lacks explicit token-level attention between Query and context tokens. QueryFormer asks: Can we introduce explicit token-level interaction during Query generation so that the Query already incorporates user, candidate item, and historical evidence before retrieving behavior sequences?

Method: From Context Interaction to Query Generation

Overall Architecture

Inputs: user/item sparse/dense features, four behavior domain sequences, time information. The model tokenizes inputs, then builds a query linking non-sequential features and behavior sequences via a query generation pipeline, followed by downstream processing and prediction.

Core flow (four steps):

Feature context update (Co-Transformer)

Candidate-relevant behavior retrieval (QuerySeqCrossAttn)

Query generation (SeqQueryCrossAttn)

Subsequent representation fusion

Notation: Nᵤ, Nᵢ = user/item non-sequential tokens; B = behavior sequences; N′ᵢ = updated item token; R = retrieved behavior summary; Q = query for reading behavior sequences.

Feature Interaction: Building Joint User-Item Context

Co-Transformer contains two interaction layers. Each layer performs self-attention within user tokens and within item tokens, then cross-attention between user and item tokens with gated fusion. Self-attention integrates same-side field information; cross-attention introduces user-candidate item relationships. After this stage, downstream modules use interacted context representations, enabling use of field combinations and user-item matching signals before behavior retrieval.

Behavior Retrieval: Adding Candidate-Relevant Evidence to Context

QuerySeqCrossAttn uses the aggregated item token as Query to attend to each of the four behavior domain sequences. Each domain outputs a candidate-relevant historical summary, added to the non-sequential token set. This provides additional evidence for Query generation: the generator can access user/item features plus retrieved candidate-relevant evidence. Interaction direction: item token → behavior sequences.

Query Generation: Behavior Summaries Actively Aggregate Context

SeqQueryCrossAttn is the core query generation mechanism. The model initializes queries from behavior sequence mean-pooling and max-pooling summaries (two queries per domain). These queries attend to the updated context tokens (Ñ = N′ ∪ R) as Key and Value via cross-attention.

For the k-th query of domain s: qₛ,ₖ = CrossAttn(q⁽⁰⁾ₛ,ₖ, Ñ, Ñ), where q⁽⁰⁾ₛ,ₖ comes from behavior summaries. Cross-attention aggregates information based on relevance between the initial query and context tokens. The final Query thus contains behavior domain information plus selected/fused user, candidate item, and retrieved evidence.

Difference from QuerySeqCrossAttn: attention direction and output usage. QuerySeqCrossAttn retrieves behavior evidence using item representation; SeqQueryCrossAttn uses behavior-initialized queries to aggregate context and form new Queries. Together they form the feature-sequence interaction flow.

Generated Queries then participate in attention reading of behavior sequences. Two lightweight processing blocks combine SwiGLU sequence encoding and token mixing to further update representations. The pipeline retains token input/output interfaces, enabling combination with other unified recommendation modules.

Multi-View Representation and Shared-Parameter Computation

QueryFormer introduces an embedding matrix with H independent tokenization columns. Same input passes through different tokenization paths to form multiple representation views, tensor shape (Bₛ, H, T, d_model). H is the number of tokenization views, distinct from multi-head attention heads.

This provides an additional capacity scaling direction : at fixed representation dimension, increasing H creates more feature representation views. The model organizes these views in a single batched parallel forward pass, then combines column outputs.

To control query computation overhead, multiple sequence queries are packed and computed with shared-parameter cross-attention, reducing duplicate Key/Value projections and GPU kernel launch overhead. Actual benefit requires joint evaluation of effectiveness and efficiency.

Experiments: Where Architectural Gains Come From

Experimental Setup and Comparison Scope

Industrial track semi-final data: ~35 million samples, 142 feature columns, four behavior domains. Train/validation split 90/10 by Parquet Row Group. Sequence length fixed at 256 (truncate older). Primary metric: AUC; also report LogLoss and inference efficiency.

Training uses MuonPlus for dense parameters, Adagrad for sparse embeddings, with EMA, mixed precision. Loss: binary cross-entropy for conversion prediction + auxiliary MAE loss on log-transformed conversion time. Local validation results reported; official test results listed separately. Note: local validation and official test sets have distribution differences, so AUC values are not directly comparable.

Comparison with Baseline Models

Under the paper's optimized training recipe, QueryFormer vs. HyFormer:

Model Configuration | Dense Params | Validation AUC
HyFormer            | 105.8 M      | 0.84477
QueryFormer, H=4    | 87.1 M       | 0.84606
QueryFormer, width 544 | 220.8 M    | 0.84631

QueryFormer (87.1M params) outperforms HyFormer (105.8M) by +0.00129 AUC with fewer dense parameters. Increasing representation dimension further improves AUC to 0.84631. This shows the architecture improves the effectiveness-parameter relationship; specific module contributions need ablation analysis.

Ablation Study: Query Generation Is Key

Fixed H=4, d_model=272, processing blocks K=2, trained 4 epochs. Best validation AUC compared:

Configuration                     | Best AUC | AUC Change
Full model                        | 0.84606  | —
Query generation → MLP            | 0.84568  | −0.00038
Remove user-item interaction      | 0.84577  | −0.00029
Remove candidate-relevant behavior retrieval | 0.84583 | −0.00023
Remove self-attention             | 0.84586  | −0.00020
Remove all four mechanisms        | 0.84496  | −0.00110

Replacing cross-attention query generation with MLP causes the largest drop , supporting the core claim that Query formation is an independently optimizable component in unified recommendation architectures. User-item interaction, candidate-relevant retrieval, and self-attention each contribute, indicating query quality and upstream context construction jointly affect prediction.

Training curves show MLP version slightly higher AUC in first three epochs but drops at epoch 4; cross-attention version achieves higher best AUC. Thus the experiment supports best-effectiveness advantage over full training, not per-epoch superiority.

Supplementary: removing conversion time auxiliary loss reduces AUC from 0.84606 to 0.84590 (−0.00016), smaller than any single attention ablation, helping separate auxiliary training objective impact from core architectural improvement.

Scalability: Representation Views, Model Scale, and Compute Cost

Benefits of Increasing Representation Views

Fixed d_model=272, H from 1 to 8: validation AUC rises from 0.84540 to 0.84615, LogLoss improves synchronously.

Effectiveness vs. evaluation cost:

H | AUC      | LogLoss   | Amortized Time
1 | 0.84540  | 0.210651  | 31.1 μs
2 | 0.84575  | 0.210432  | 34.5 μs
4 | 0.84606  | 0.210309  | 43.1 μs
8 | 0.84615  | 0.210225  | 58.7 μs

H=1→4: AUC +0.00066, time +39%; H=4→8: extra gain only +0.00009. Paper selects H=4 as practical configuration balancing effectiveness and inference constraints. Overall, H=8 amortized time ~1.89× H=1; cost increase is sublinear due to packed execution.

Different Roles of Width, Depth, and Training Epochs

Fixed H=4:

Width d_model 272→544: AUC 0.84606→0.84631, dense params 87M→221M. Width increase improves effectiveness but with significant parameter growth.

Processing blocks K=1→3: AUC 0.84598→0.84616, gain +0.00018.

Training epochs 4→11: AUC +0.00010, but LogLoss worsens from 0.210309 to 0.210712. Longer training improves ranking but not probability calibration.

These results indicate capacity and training budget must be jointly allocated across multiple dimensions. For this data and configuration range, multi-view is an effective scaling direction; further comparison of scaling strategies requires joint examination of parameters, time, and prediction metrics.

Comparison Across Parameter and Data Scales

Data scale experiments compare 87.1M QueryFormer vs. 105.8M HyFormer. Within the paper's parameter budget range, QueryFormer achieves higher validation AUC at similar dense parameter scales; across ~20% to full training data, tested QueryFormer configurations maintain advantage. These results support the architecture's parameter utilization efficiency and performance at different data scales. Scaling fits are empirical within tested intervals; extrapolation to larger scales is not claimed.

Official Competition Results and Post-Competition Extensions

QueryFormer won industrial track with test AUC 0.83254 . Post-competition increased-parameter configuration achieved 0.832713 (+0.000173), a post-hoc experiment.

Final submission used full-data training and multi-teacher to single-student distillation. Paper reports these contributed ~0.0001 and ~0.0007 AUC gains respectively; distillation inference uses a single student model. Thus the official score reflects the combined architecture, training recipe, and final training strategy; judgments on specific mechanisms like query generation should rely on corresponding ablation experiments.

Discussion and Conclusion

QueryFormer treats query generation in unified recommendation modeling as an independent architectural design problem. By first updating user-item context, then retrieving candidate-relevant historical evidence, and finally letting behavior-initialized queries aggregate context, the model forms a unified processing pipeline connecting multi-field features and behavior sequences. Ablation supports cross-attention query generation effectiveness; multi-view experiments demonstrate another capacity scaling dimension and its compute cost.

Experiments are primarily on one large-scale competition dataset; multi-view configurations increase parameters and runtime. Current results do not cover cross-dataset generalization, different hardware conditions, or systematic evaluation when integrated into other backbone networks. The structural token input/output interfaces provide compositional basis; broader applicability requires further validation.

For unified recommendation architecture research, this work suggests: beyond focusing on sequence encoding and feature interaction operators, one can systematically study how the query representation connecting them is formed. Combining module ablation, parameter and data scaling, and inference efficiency evaluation enables more complete analysis of this design's benefits and boundaries.

References

Yuanzhe Zhou, Zhaoyang Zeng. QueryFormer: Winning Solution for KDD Cup 2026 Tencent UniRec Challenge. arXiv:2609.16548, 2026.

Tencent UNI-REC Challenge: Scalable Recommendation Systems.

https://www.linkedin.com/posts/tencentglobal_ai-machinelearning-recommendationsystems-activity-7442133255124062208-Uhd2

TAAC official website: https://algo.qq.com/ TAAC2026 sample data: https://huggingface.co/datasets/TAAC2026/data_sample_1000 Yunwen Huang et al. HyFormer: Revisiting the Roles of Sequence Modeling and Feature Interaction in CTR Prediction. 2026.

Mingyang Liu et al. EST: Towards Efficient Scaling Laws in Click-Through Rate Prediction via Unified Modeling. 2026.