Why Strong Models Still Fail in Production: The 'Ontology First, Agent Second' Rule

Conference Theme: Data First, AI Real

DACon 2026 Beijing, guided by the Shanghai Key Laboratory of Data Science, ran for two days with one main forum and 17 sub-forums. DataFun founder Wang Dachuan framed the theme "Data First, AI Real" as eight Chinese characters: "本体到位，Agent上岗" (Ontology ready, Agent on duty). The main forum was structured as five progressive questions: why deployment is hard despite strong models, what is missing for agents to enter core workflows, what constitutes the ontology, how to build and maintain it, and what ROI it delivers in industry.

Opening Keynote: Xiao Yanghua — Why Ontology Became Consensus in One Year

Fudan University professor Xiao Yanghua (via video) contrasted last year's skepticism with this year's consensus, giving three reasons:

Knowledge engineering is perpetual. Large models change knowledge form but not its importance; "knowledge becomes even more critical" for deployment.

The last mile is filling private knowledge blind spots. Models learn open, world, and domain knowledge but not enterprise private knowledge; supplying that context turns a model into an industry expert.

Parametric knowledge is inherently probabilistic. "As long as knowledge is trained into the model, its generation is a probabilistic process." For 100% reliability, symbolic knowledge remains necessary. His counter-intuitive advice: "Don't let the large model memorize every enterprise's stuff." Models should focus on understanding cross-industry logic; rules and scripts are controllable and verifiable.

He also outlined the ontology–model relationship: they complement each other — models accelerate ontology construction, while ontology governs model uncertainty. He recommended a bottom-up path: deploy hundreds of digital workers across thousands of positions, distill knowledge, then grow into department-, enterprise-, and industry-level ontologies. Selection principle: "Knowledge that can be transferred or shared will likely be learned by foundation models. What we truly need to retain is the non-transferable, non-shareable knowledge."

Session 1: Hu Haoyuan (JD Health) — Benchmark Leadership ≠ Delivery Capability

JD Health's medical LLM scored 72.3 on the senior physician exam, ranking first on First Bench Hard, Total, and MedBench. Yet Hu emphasized: "Model leadership on benchmarks does not guarantee correct answers." Medicine needs materials a doctor can use next, with white-box transparency — "you cannot be a black box; doctors must be able to intervene at any step."

Core concept: "Completion conditions." "We write reverence for life into the system's completion conditions." Single-step accuracy near 99% hits a ceiling; long-horizon tasks of 500+ steps explode in complexity. Solution: formalize verifiable, recoverable steps.

Abandoned RAG for deterministic retrieval. Medical trust cannot rest on probabilistic retrieval. They built offline structured knowledge bases and replaced retrieval with deterministic MedViki Joint Search — paths are explainable, errors locatable and fixable.

Chain-of-thought can deceive; code cannot. CoT often rationalizes decisions post-hoc, producing "professional-feeling" not "professional-capable" models. They enforce output of every operation and code step: "Code cannot lie — it either executes or fails." Optimization targets the whole system (operators, tools), citing "Learning Beyond Gradient."

Results. Multi-expert distillation reduced to one model serving all business, cutting invocation volume ~70%. Inpatient note acceptance without edits: 75%. Multi-disciplinary consultation consistency: 70%. Most complex task: 800 autonomous loops, 500 tool calls. "Models can be procured, but how to accept a real task after it runs — that you cannot buy."

Key takeaway: "Completion conditions" are formalized business rules, judgment criteria, and delivery standards. They live scattered in systems and experts' heads — who organizes them into agent-readable form?

Session 2: Liu Bin (Baidu Intelligent Cloud) — Agents Need a Business Map to Enter Core Workflows

Liu Bin argued the biggest shift is decision mode: from "humans decide, machines assist" to "agents decide and execute, humans set goals and guardrail." Metrics shift from compute/token consumption to business ROI and P&L impact.

Two gaps: (1) Effectiveness — "No matter how smart, the model cannot write your business logic" (like a bright grad who doesn't know how to reorder when stockouts occur). (2) Trust — "If it says order 100 units, can you believe it? It must explain what data, permissions, and logic it used."

Ontology as enterprise world model — three layers:

Business object map (orders, inventory, logistics, product relationships)

Business logic (how to reorder on stockout, how returns flow — "often lives in experts' heads")

Externally readable and writable — "If it's only readable, it's not a true production system… you dare write, but must be able to roll back." AI guesses at legacy tables; ontology gives humans and agents a single business language.

Systems engineering: let agents "iterate" in branches. Core layer borrows OS ideas: unified metadata, unified file interfaces (humans and agents see same PPT/Word), fine-grained permissions, abstract Spark/Flink/Ray as process interfaces. Most critical: full-chain Git-ification — from pipeline, metadata, ontology to agent, instant branching allows safe experimentation: "can branch, so can iterate fearlessly; merge if right, discard if wrong."

Two cases:

Supply chain shortage: tens of thousands of materials, few experts. Ontology of materials, inventory, orders, logistics + expert logic distilled → zero missed shortages, decision adoption >95% ("not 100% because latest info still in experts' heads").

Engineering change: thousands of parts, one change. Previously reading massive docs, prone to omissions. Ontology-first → expert report generated, expert confirms.

Session 3: Du Junping (DataStrato) — Ontology Is Three Pillars, Not One Graph

DataStrato founder Du Junping reframed the problem: as agents move from answering to reasoning/decision to API invocation/workflow triggering, context requirements escalate. He defined three interdependent pillars:

Unified metadata. Answers "where is data, how to access" — stable identity connectors and unified access for databases, warehouses, lakehouses, streaming, files, model assets. Cross-domain value: narrow search from thousands of tables to dozens or fewer. Gravitino now a de facto standard like Spark/Flink.

Open semantic layer. Answers "what does this data mean in business" — metrics, dimensions, grains, definitions. Example Net Revenue: semantic model specifies calculation expression, time/analysis dimensions, joins between order lines and products. "Agent doesn't guess fields or redefine revenue grain." Open Semantic Interchange (OSI) solves tool interop; Gravitino adds runtime: definitions, permission checks, asset binding.

Ontology. Handles not "compute a number" but "identify a business concept." Query "which tables contain customer PII, any leakage?" requires relationship graph — privacy radiates to email, phone, address entities. Binding concepts to relationships to fields enables graph-based scope queries, not scalar/vector lookups.

Context governance — four checks: who can view/use/modify semantic model; underlying table/field permissions ("model definition right ≠ automatic data access right"); business conditions; execution and audit.

Top-down vs bottom-up: Top-down from apps — fast, business engages, but definitions trapped in single app, short-term success doesn't become enterprise asset. Bottom-up unified catalog/governance — shared foundation, but slow; "technical architecture cannot replace business judgment" — metadata shows many order tables but not which grain defines revenue. Answer: combine both.

Session 4: Guan Tao (Yunqi) — Ontology Is One Knowledge Modeling Form Among Five

Guan Tao deconstructed "knowledge": past 20 years solved "data → information" (RDBMS, big data turn raw logs into BI reports, "but report interpretation still needs humans"). AI now enables "information → knowledge" via understanding, extraction, generation, reasoning.

Ontology is just one modeling form. Same fact: text ~100 bytes, table 10-20 bytes with precise compute, screenshot bloats, also vector, graph, parameters. "No single form rules all" — models hold everything but lack explainability and cheap reasoning; tables are accurate and readable but require curation and lack flexibility.

Five industry routes:

Palantir: full modeling + action layer, validated in defense, extreme cost (hence FDE on-site).

Snowflake Semantic View: models only structured data, wins on accuracy/determinism, typical Chat BI / Data Agent starting point.

OSI (Open Semantic Interchange): DBT-led, multi-vendor, separates semantics from ontology, allows GMV/revenue different grains mapping to same semantic.

Databricks hybrid: semantic layer, graph, document search mixed.

Agent Memory systems: almost no modeling, relies on directory structure + text retrieval, very successful in personal knowledge.

Conclusion: selection "should not glue to one modeling form."

Knowledge acquisition: documents easy, human brains hard. "Hardest is in people's heads." Their solution: tracing — humans continuously emit knowledge during agent interaction. Salesperson asks "adjust this month's GMV", agent returns 12M; salesperson corrects "this metric includes refunds", that sentence captured as metric correction. "This knowledge in human brain, can't write doc, but this way works."

Two empirical validations:

Client with 1,000 tables needed semantic views. Manual writing inefficient and inaccurate. Reverse-distilled from existing BI reports → accuracy >90%, "won't hallucinate missing metrics."

Top content platform with excellent docs but metric accuracy only 30+%. Problem: knowledge overload — multiple similar chunks retrieved, attention scattered across selection, SQL generation, verification, attribution. Distilled into metric system → accuracy nearly doubled, tokens reduced. "Knowledge distillation is entropy reduction: data shrinks, knowledge shrinks, but accuracy rises."

Four architectural consensuses:

Dev phase: agents for rapid iteration; production: swap to deterministic components.

Adopt 2024 Databricks/Snowflake "medal architecture" as knowledge production pipeline base — two isomorphic pipelines, unified metadata, AI as new engine, incremental not one-off.

Knowledge platform not one-off delivery; "no fixed test set" — shift from TDD to EDD (evaluation-driven development). "Even if built well today, half year later it likely unusable."

Session 5: Feng Xingzhi (Kaos/COSMOPlat) — Industrial AI: No General Model, Only "One Meter Wide, Hundred Meters Deep"

Kaos VP/CTO Feng Xingzhi framed industrial reality with two coordinates, then calculated ROI.

Five dilemmas: Industrial data lacks unified standards (protocols, quality, silos are engineering problems); true knowledge in veteran workers' heads; hallucination unacceptable — "99% pass rate not enough"; must do large-small model collaboration + physics/mechanism constraints; precision via mechanism models and evaluation; governance via regulations; value must shift from pay-per-function to pay-per-outcome.

No general industrial model — only "one meter wide, hundred meters deep." Industrial internet since 2013 as disruptor but "never solved replication at scale" — 0-to-1 works (Predix, MindSphere), scaling fails. Visual inspection example: one line POC great, replicate to dozens of lines and all steps, cost untenable — "project 500k, customer demands 20k per process." "Industry may have no general large model; it needs vertical large models, even domain-specific large models."

Platform & assets: Kaos born from manufacturing (Haier, 160 factories globally); 4,700+ industry base models and agent dev tools; converged to five verticals (appliance, semiconductor, equipment, auto, process chemical) covering 40+ scenarios; ontology side: 500+ ontology model libraries.

FDE (Field Delivery Engineer) role critical difference: "Our FDEs are not for platform staff, but for business users." Manufacturing needs often vague; FDE value: multi-round interaction to refine requirements, directly extend new function node on customer's existing product, so delivery lands back on product.

Scenarios & numbers:

Operations analysis: margin anomaly → ontology retrieves objects, structural/volume/raw-material cost decomposition models diagnose price vs mix issue.

Factory layer: digital plant manager owns profit; base MES/WMS/QMS/IoT + digital twin simulation.

Equipment management: fault root-cause reasoning + spare parts ("not more stock, but right stock").

Flexible scheduling: self-owned factories changeover 10x/day, each saves 30%, ~5M RMB/year.

Energy: compressor station unmanned, scheduling optimization adds 5-10% efficiency.

Synthesis: Ontology from semantic web 20+ years, value target always "data interoperable, computation accurate, trustworthy, replicable." "It's a continuous approximation, not a single answer." Solving industrial AI's determinism requires "not one team enumerating all" but commercial and development ecosystems co-building.

Synthesis: Five Sessions Plug Four Gaps on the Agent Deployment Path

Viewed together, each session plugs a specific vulnerability:

Session 1 plugs the acceptance gap : without formalizable completion conditions, agents just run long, not right.

Session 2 plugs the understanding & permission gap : without business objects, logic, and rollback-capable write rights, agents only guess, no one lets them act.

Sessions 3 & 4 plug the supply gap : without stable three pillars and maintainable, production-grade knowledge, agents work today, fail in six months.

Session 5 plugs the value gap : ontology must convert to money in changeover, inspection, energy.

Commonality: none of these gaps live in the model. This is the practical meaning of "Ontology ready, Agent on duty" — ontology is not re-storing enterprise knowledge, but turning enterprise objects, logic, boundaries into something both humans and agents can read, execute, and roll back. Agent on duty is not swapping a stronger model, but connecting this fabric to the execution chain.

Re-reading the eight characters backwards: not "when can agents go on duty" but "when can ontology get ready." The former depends on model iteration speed; the latter on whether the enterprise is willing to articulate its business clearly.

Three resulting shifts:

Evaluation standard: from "benchmark-oriented" to "delivery-oriented." Hu's 72.3 and #1 rank only prove base adequacy; real metrics: 75% no-edit adoption, 800-step loops, zero missed shortages, 30% changeover savings.

Core asset: from "model & prompts" to "ontology & context." Liu Bin: business object map, logic, read/write. Du Junping: three pillars + governance. Guan Tao: log-to-pipeline-to-knowledge-expression engineering system. Different phrasing, same verdict: models rotate, ontology and traces compound.

Organizational model: from "IT owns everything" to "business builds, platform governs, FDE accompanies." Liu Bin: platform users are now agents. Feng Xingzhi: FDEs serve business. Guan Tao: knowledge platform must be pulled by real agent tasks.