AI Generates One‑Shot Semantic Layers: MotherDuck Finds Eval More Valuable Than the Model

In the past year the semantic layer has re‑emerged as a hot concept in data and AI because agents need a business‑level language to understand concepts such as “active customer” or “net revenue”.

MotherDuck conducted an experiment around Malloy, an open‑source data‑modeling language, to test two hypotheses: (1) an agent can, with minimal human effort, automatically construct an accurate Malloy semantic layer using a data manual, training Q&A, and schema; (2) the semantic layer would reduce prompt tokens, speed up responses, and keep accuracy comparable to a baseline of Markdown + SQL.

The first hypothesis was confirmed. Using the public experiment repository, the model read the manual, training questions, and schema, generated a Malloy model, then compiled, executed, and repaired it in a loop. Provenance and hash checks ensured the model‑generated layer, not a manually tweaked one.

The second hypothesis was disproved. On the DABStep benchmark (419 held‑out questions), the results were:

Markdown + SQL (Gemini): 418/419 correct (99.8% accuracy), median prompt tokens = 42,370.

Malloy (Gemini, fixed model): 295/419 correct (70.4% accuracy), median prompt tokens = 106,609 (≈2.52× more).

Malloy (Sonnet + Opus, best official run): 382/419 correct (91.2% accuracy), median prompt tokens = 84,010.

These numbers show that the additional semantic layer did not reduce the context the agent needed; it actually increased token consumption.

Further analysis revealed that agents often bypass the pre‑built semantic layer. In the best Malloy run, 56.6% of questions were still answered via the raw SQL path, only 12 of 83 defined Views were ever used, and 52 Views were never referenced.

The report explains why a pre‑constructed semantic model can limit an agent: for example, a “most expensive fee” query used an average calculation in the semantic layer, while the correct answer required a sum, causing the agent to propagate the wrong logic even after falling back to SQL.

This leads to the insight that the semantic model is better viewed as a governance contract, while the business intent, context, and evaluative tests (Eval) become the more valuable, repeatable assets. MotherDuck argues that “your data tests become more valuable than your semantic layer”.

MotherDuck still cites three irreplaceable uses for semantic layers: isolating downstream consumers from schema changes, propagating unified metric definitions, and providing deterministic guardrails for domain experts.

Overall, as code generation becomes cheaper, the most valuable asset may shift from the implementation (the semantic model) to the desired system behavior captured in evaluative tests and business intent.