Airbnb CASTLE: Synthetic Queries & Labels for Cold-Start Natural Language Search

Introduction

Natural language search lets guests express needs in their own words — e.g., "romantic cabin near ski slopes" — without navigating filters. Airbnb needed this capability to work well from day one, but that meant training models before any real user queries existed and having relevance labels to map queries to listings. Both query and label data had to be synthesized from scratch: the cold-start problem for natural language search.

Prior approaches separate query generation from label assignment. InPars prompts an LLM to generate queries from a single document, then relies on another LLM judgment for relevance labels, introducing a second error source. Promptagator uses few-shot examples to improve query diversity, yet its synthetic distribution still diverges from real user behavior.

Challenge

Useful synthetic training data must match the distribution of real guest searches: phrasing preferences, query length, and frequently mentioned attributes. The article measures this gap with Kullback–Leibler (KL) divergence between synthetic and real query distributions, comparing both surface statistics and attribute-type distributions (listing type, amenities, location). Previous methods scored KL divergences between 9.33 and 26.2 — far from real users. Beyond distribution alignment, relevance labels are equally critical: knowing which listings match a query is the other half of the problem.

Solution: CASTLE Framework

CASTLE (Contrastive And Seed-guided Training for natural Language sEarch) jointly generates queries and relevance labels using de-identified booking sessions. When a guest browses multiple listings and books one, that choice implies a preference: the booked listing is preferred over the others. CASTLE forms (positive, negative) listing pairs from these sessions, with negatives randomly sampled from viewed-but-not-booked alternatives. Listing features include amenities, details, and a one-sentence summary of recent public reviews, all stripped of PII. An LLM is prompted to generate a query that the positive listing satisfies but the negative does not, so relevance labels are guaranteed by construction — no post-hoc judgment needed. Crucially, the positive/negative split comes from real historical booking behavior, not the LLM, avoiding circular reasoning.

Query Generation

Three prompt variants maximize coverage and realism:

Template-based prompts ensure structural diversity for common query patterns.

Few-shot prompts use ~500 seed queries collected from targeted user research as linguistic exemplars, grounding generation in actual guest phrasing.

Attribute-based prompts focus queries on distinguishing attributes between the two listings (e.g., fireplace, proximity to ski resort).

An explicit diversity mechanism varies query length and phrasing across samples to prevent "query collapse" — the tendency of unconstrained LLMs to converge to the same short phrase. Together, these variants produce queries whose word-count distribution has standard deviation σ = 2.48, closely matching real guests' σ = 2.49. Baselines without seed guidance (InPars, Promptagator) yield σ = 0.99. Real Airbnb guests search with concise, intent-focused phrases of 4–8 words, not long sentences; CASTLE targets this real distribution.

Flexible Input Design

For listings lacking behavioral contrast pairs (e.g., new listings with no booking history), CASTLE introduces a Virtual Judge (VJ) — an LLM-based evaluator that assigns relevance labels to additional query–listing pairs. This extends coverage to long-tail intents without sacrificing the quality guarantee of the core contrastive construction. A variant called CASTLE-R uses real queries as seeds once production traffic begins to accumulate.

Results

CASTLE is compared against InPars-style generation and Promptagator on two dimensions: query distribution alignment and label quality.

KL divergence (query distribution): CASTLE achieves 1.01 , a 9.2× improvement over the best baseline (Promptagator at 9.33).

Attribute-type KL divergence: CASTLE scores 0.08 , even lower than the human-curated research seed queries (0.09), meaning generated queries reflect real guest intent diversity more faithfully than the seeds themselves.

Human evaluation: three independent domain experts reviewed 200 sampled triplets. All three annotators agreed with CASTLE's relevance labels at 91–93% , confirming that contrastive construction yields high-quality labels without any manual annotation in the generation pipeline.

Even after cold-start ends and real traffic accumulates, CASTLE's synthetic data remains valuable. Because generation is conditioned on concrete listing features, it naturally oversamples long-tail intents that rarely appear in real traffic — e.g., listing types or amenity combinations guests search for occasionally but that are critical when they do.

Conclusion

CASTLE's core insight: real booking behavior and LLM generation capabilities are complementary, not competitive. Contrastive listing pairs from booking sessions ground query generation in actual platform behavior and provide relevance labels during construction; LLM prompting adds query diversity and linguistic variation that behavior data alone cannot supply. Together they close the gap between synthetic and real user distributions in a way neither can achieve alone.

Future work explores multi-turn conversational search (e.g., "cabin near ski slopes" → "must allow dogs"), where cold-start data generation is more challenging but the same listing-contrast approach may extend. Full technical details in the CIKM 2026 paper: arxiv.org/abs/2605.21812 (DOI: 10.1145/3799682.3841071).

References

[1] Jeronymo, Vitor, et al. "InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval." arXiv:2301.01820, 2023.

[2] Dai, Zhuyun, et al. "Promptagator: Few-shot Dense Retrieval From 8 Examples." ICLR, 2023.

[3] Wei, Wendy Ran, et al. "CASTLE: Contrastive and Seed-Guided Training for Cold-Start Natural Language Search." ACM International Conference on Information and Knowledge Management (CIKM), 2026. DOI: 10.1145/3799682.3841071.