How Airbnb Built an Agent Harness to Make AI Research Reproducible and Scalable

Unstructured Data Exploration Challenges

In 2025, Airbnb prepared to launch an AI customer service assistant. Before launch, they needed to understand the scenarios it would encounter, including rare but risky events. This required classifying these events and their proportions to build datasets for a more responsible product. The research was rigorous but highly manual: finding data, reviewing samples with experts, generating representative datasets took months of close collaboration and iteration. Methods were scattered across notebooks, spreadsheets, and documents, relying on individual judgment and manual maintenance. For a single study this was acceptable, but scaling to weekly research across new languages, regions, and LLM applications demanded a reusable methodology.

Insight Miner: Agent Harness for Unstructured Text Understanding

Insight Miner provides engineering capabilities for querying, large-scale annotation, embedding generation, clustering, and tracing so users can perform large-scale analysis without learning new infrastructure. Its core pipeline — extract, embed, cluster — is mature. Around this pipeline, Airbnb integrated proven practices from internal and external research: prompt tuning, hard-case mining, contrastive annotation, and an analysis path that starts with unsupervised exploration then shifts to classification. Previously these methods lived in separate notebooks, shared by copy-paste; now they reside in a shared toolkit that absorbs improvements from each study.

A user poses a research question in conversation, launching a study. The agent acts as research partner, executor, and framework expert. Insight Miner is not limited to specific datasets, domains, or questions; it analyzes unstructured text from various sources, selects analytical perspectives based on the research question, and applies the same rigorous method every time.

When expanding the AI customer service assistant to more languages and countries, Insight Miner reduced research time from months to days. Its greater value is raising rigor and speed simultaneously without trade-offs. Data scientists shift focus from executing analysis to improving methods at each step. Mechanical work scales and parallelizes, freeing time for careful judgment on the hardest-to-interpret or most critical data, validating hypotheses and groupings, and building solid qualitative and quantitative understanding of data and product. Rather than automating analysis away, the framework brings human judgment deeper into the most critical analytical steps.

From Technical Teams to Broader Business Teams

Initially designed for data scientists, Insight Miner quickly spread. Within a year, dozens of teams used it for hundreds of research types. Non-technical users now outnumber technical users, especially in operations and product insights. A more accessible UI for data exploration and agent conversation lets more domain experts participate. Experts who never wrote code can now run large-scale analyses directly, without waiting for limited engineering or data science support. Projects that previously lacked resources or relied on manual review of hundreds of samples can now apply shared best practices to analyze data orders of magnitude larger. Use cases include coding open-ended survey responses, evaluating model performance, and analyzing fraud patterns.

A New Class of Infrastructure

Agent harnesses like Insight Miner are full-stack systems and a new class of infrastructure requiring continuous development, maintenance, evaluation, and improvement — work well-suited for assistance by other agent systems. For example, separate agent systems help adapt instructions for newly released models, incorporate new best practices, inspect real-world usage to find pain points, and monitor and reproduce emerging defects with fix proposals. Together they form a larger agent collaboration environment reshaping daily work in data science and beyond.

Such frameworks apply wherever domain experts hold methodology worth codifying: legal review, policy analysis, any field treating research method as equally important as the answer. They are key to shifting knowledge work to an AI-first mode. Airbnb's CTO has written that as models commoditize, sustained value comes from proprietary data, deep workflow integration, and professional judgment (taste). Agent harnesses accumulate all three: expert judgment becomes methods adoptable by every team; each new user deepens workflow integration; feedback loops are built in so every run can improve the framework for everyone.