How Three Companies Built AI Systems That Know When They Can't Be Trusted

Amap: Causal Inference Engine for Marketing Subsidy ROI

Xu Xiaowei (Amap, Senior Data Technology Expert) presented 《高德营销补贴 ROI 评估：从人工一次性到可信决策引擎》 . Marketing subsidy ROI is fundamentally a causal inference problem: how many incremental orders did the subsidy generate? The core difficulty is the counterfactual is unobservable — you can never know whether a user would have ordered without the subsidy. Because there is no ground truth , a stronger model becomes more dangerous: it will always produce a plausible-looking ROI number.

Amap made two key architectural choices:

Method selection: PSM + DID instead of simple before-after or pure A/B. Before-after conflates natural growth with subsidy effect; pure A/B is costly, slow, and campaigns contaminate each other.

Where to place AI: "Observable error vs. silent error" criterion. If an error surfaces immediately (e.g., a miscalculation that can be verified), that stage can use AI. If an error silently pollutes the conclusion (e.g., wrong metric definition yields a number that looks fine but means something else), that stage must use deterministic rules.

This led to a dual-brain architecture : a frozen kernel handles deterministic computation; an Agent Loop handles orchestration; a parity contract guarantees reproducibility. Three gates sit on top:

Credibility grading — different evidence strengths map to different conclusion levels.

Caliber gate — non-compliant metric definitions are blocked before computation.

Multi-method triangulation — multiple causal methods cross-validate; inconsistency triggers downgrade.

The goal is not "calculate more accurately" but let the system judge whether it is qualified to conclude — i.e., teach it to say "I'm not qualified to conclude." The engine now runs hundreds of campaigns daily across ride-hailing and local-life businesses. It has caught three real incident classes:

ROI inflation — caused by caliber pollution.

Negative increment misjudgment — overlapping campaigns causing attribution contamination.

False significance — small samples lacking covariate overlap.

All three cannot be fixed by "a bigger model." Amap openly admits unsolved parts: credibility grading still relies on partly manual thresholds; ~15% of campaigns fall in a "assessable but evidence-insufficient" gray zone; when multiple causal methods disagree, there is no accepted offline selection standard — currently resolved by triangulation + human arbitration.

"In decision systems without ground truth, the engineering focus shifts from 'pursuing higher accuracy' to 'managing the system's cognition of its own correctness.'"

JD Retail: From Conversational Query to Organizational Data Workspace

Wu Xujing (JD Retail, Algorithm Engineer) presented 《从对话式问数到组织级数据工作空间：京东广告数据分析 Agent 的工程化实践》 . After conversational query (Text-to-SQL) works, the next problem emerges: early agents could answer but were hard to reproduce, reuse, collaborate on, or audit. A successful analysis living only in a chat transcript equals never having happened for the organization.

JD's path: Graph-first, Workspace-light . They describe the relationship as: Graph is the verb (execution process), Workspace is the noun (execution result).

Graph — decompose query, compute, chart, report, subscription, attribution into an orchestratable execution graph; first punch through the full ad-analysis loop.

Workspace — lightly persist Query, Dataset, Report, Template, Execution, Provenance.

Why not build a full Workspace first? Heavy platform investment risk; real paying signals not yet validated. Use Graph to penetrate high-frequency scenarios first, then decide whether to build out the platform — a deliberate sequencing.

Core design shift: from Conversation-centered to Artifact-centered :

Definition / Execution / Result separation — Template/Workflow = definition; Execution/Task = execution; Artifact = result.

Stable artifact naming and versioning — Query, Dataset, Chart, Report, Code, Template each have stable identities.

Provenance evidence chain — a report can trace back to the exact query, data, code, metric definition, and template version.

Capability and Skill decoupling — business capabilities serve multiple Agent Runtimes, not bound to a single page or single Agent.

Four production capabilities delivered: intelligent query, auto-report, template subscription, anomaly attribution. Template subscription is the organizational loop : power users codify methods; regular users one-click consume stable reports — "producers produce methods, consumers reuse methods."

A notable engineering detail: error and retry design locates the exact node, parameter, data source, and next action — most query products just say "query failed."

JD provides a concrete checklist for whether to invest further in Workspace — five buying signals: save, reuse, trace, share, template consume . If these five actions don't happen organically, the Workspace is self-indulgence.

Xiaoying Technology: Four-Stage Evolution from Purchased Bot to Agentic Voice Agent

Wu Wenbin (Xiaoying Tech, Model Application Development Lead) presented 《从外购到自研，从向量 RAG 到 Agentic 语音智能体——企业对话机器人的完整演进之路》 . A complete build-vs-buy retrospective across four stages:

Third-party platform trial — discover limitations.

Vector knowledge base + RAG — establish baseline dialogue capability.

LLM integration + standardized SOP process bot — stabilize deterministic flows.

Upgrade to Agentic architecture — autonomous task-planning voice bot.

Why self-build? Purchased bots suffered from hard customization, high long-term calling cost, and inability to deeply integrate with business processes — the last being a capability ceiling.

The stepwise path matters: many teams stall at stage 2 or jump straight to stage 4 and fail. Three hardest production challenges and their solutions:

Real-time voice latency — audio streaming, vector retrieval, serial LLM calls chained together exceed timeouts. Solution: module async decoupling, hot knowledge-base pre-caching, tiered inference strategy .

Agent process runaway — complex flows cause deviation from standard SOP, hallucinated outputs. Solution: soft/hard rule dual constraints, task state persistence, multi-round behavior verification .

Knowledge-base hallucination — fatal in customer service. Solution: layered memory and task scheduling system .

The "soft/hard rule dual constraints + task state persistence + multi-round behavior verification" mirrors Amap's philosophy: draw a visible line for the Agent's freedom instead of hoping it won't drift.

Results: significant uplift in human-machine dialogue completion rate; long-term service cost reduced — directly addressing the original self-build motivation. Future roadmap: multi-agent collaboration, continuous inference cost reduction, automated evaluation system construction — the third item is especially key because the first two sessions left "how to judge if it got better" unsolved.

Three Converging Threads Across the Customer Journey

The three sessions map to three journey stages: Marketing spend (should we spend?) → Ad analysis (how did it perform?) → Customer service (how do we retain?) . Methodologically they converge on three shared principles:

1. Core is not "make AI answer better" but "make AI know whether it can answer"

Amap: credibility grading + caliber gate + triangulation → system actively downgrades when evidence is insufficient, says "I'm not qualified to conclude."

JD: Provenance evidence chain → every conclusion traceable to query, data, definition, template version; untraceable = doesn't count.

Xiaoying: soft/hard rule dual constraints + task state persistence + multi-round verification → Agent locked near standard SOP.

2. Turn one-off outputs into sustainable assets

Amap: turned manual one-off evaluation into an engine running hundreds of campaigns daily .

JD: turned a chat Q&A into reproducible, reusable, traceable organizational assets with a producer-consumer method loop.

Xiaoying: turned a purchased bot into a self-evolving technical line they own .

Shared judgment standard: after the project, the organization must have more capability than before. If AI just swaps labor cost for compute cost with no capability沉淀, it's not worth it.

3. Pragmatic cost/latency trade-offs — no hand-waving

Amap: credibility grading still uses manual thresholds; 15% gray zone; method ranking not automated.

JD: explicitly refuses heavy Workspace until buying signals appear.

Xiaoying: self-build driven by long-term cost; roadmap includes "continuous inference cost reduction."

All three openly acknowledge unsolved problems. The DACon 2026 Beijing session (Oct 23-24, Beijing Hilton Yilin) will cover these three talks plus the unresolved parts.