Agentic Lakehouse: Rethinking Data Architecture for AI Agents

Background: Why Traditional Lakehouse Must Evolve for Agents

Traditional lakehouse users are analysts, data engineers, ML engineers, and business users who interact via SQL, pipelines, and BI tools. AI agents change this by autonomously selecting tools, calling data services, combining multiple queries, and using results for analysis or business actions. This demands four new capabilities:

Discoverable and understandable data: Agents need business meaning, metric definitions, table relationships, freshness, and applicability — not just technical table and column names.

Machine-friendly access: Structured tool interfaces (APIs, programmatic clients, MCP) with parameter constraints, error handling, timeouts, retries, and call logs.

Explicit freshness and consistency: Real-time risk control needs fresh data; financial analysis needs consistent historical snapshots. Not every agent requires sub-second latency.

Governance for automation: Agents may chain tools and retry on failure. Access control must extend beyond human workflows; shared service accounts with excessive permissions create audit gaps.

Therefore, an Agentic Lakehouse is not merely adding a chat interface but redesigning discovery, semantics, access, governance, and performance optimization for machine users.

Core Concepts and Reference Architecture: Four-Layer Structure

Terminology Clarification

Dremio uses Agentic Lakehouse emphasizing semantic layer, governed access, agent interfaces, and automated optimization. Alibaba Cloud OpenLake uses "Agentic Data Architecture" focusing on unified metadata and multi-engine access. Apache Fluss 1.0 positions itself as real-time data infrastructure for AI workloads. These are distinct vendor positions, not a unified standard.

Working definition: Agentic Lakehouse — on top of open lakehouse, add AI-agent-oriented semantic understanding, machine interfaces, governed queries, and automated operations so agents can access and analyze enterprise data under explicit permissions and data contracts.

Four-Layer Reference Architecture

The layers express architectural responsibilities; not all products deploy identically, nor does every component inherently possess all shown capabilities.

Why Iceberg, Catalog, and Query Engine Must Be Separated

Table format (e.g., Iceberg, Paimon, Lance) defines data organization, metadata, snapshots, and data files.

Catalog handles table discovery, metadata management, and access entry points.

Query engine / data service executes SQL, enforces query permissions, and reads actual data.

Example: Unity Catalog provides access control, row filters, column masking, but policy execution depends on version and compute environment. Credential vending (short-lived credentials) is implemented by some catalogs/platforms, not a universal requirement. Production design must clarify whether access uses short-lived credentials, engine service identity, or other controlled proxies.

MCP's Real Place in Architecture

MCP (Model Context Protocol) connects models to tools and resources, enabling agent discovery and invocation. It does not decide whether a principal may access specific business data. Correct responsibility split:

Agent Runtime: task planning and tool selection.

MCP Server / tool interface: exposes callable capabilities.

AuthZ/authN and policy enforcement: verifies caller and access scope.

Query engine / data service: reads actual data.

Audit system: records principal, tool, parameter scope, and result.

MCP is not a security boundary replacement and cannot serve directly as data authorization.

Apache Fluss 1.0 for Agent Scenarios

Fluss 1.0 released 2026-09-21. Key capabilities: Gateway, native Python client, data lifecycle management, real-time and historical data collaboration. Below, product capabilities are separated from architecture capabilities requiring custom integration.

Gateway: Lightweight HTTP Entry, Not a Complete Agent Query Service

Fluss Gateway (Rust, reuses Fluss Rust client core) provides REST interfaces, positioned as Preview. Capabilities: metadata discovery, table management, batch append, upsert, partial update, delete writes. Example: HTTP write events to Fluss log table; with Lakehouse support and running Tiering Service, data flows to lake tables with minute-level freshness (not a fixed latency guarantee).

Three critical limitations:

Preview stage — API and config may evolve.

No record read interface; no primary-key point lookup, prefix scan, or log scan.

No HTTP caller authentication, TLS termination, or end-user identity propagation; downstream Fluss uses Gateway's shared service identity.

Thus Gateway cannot be described as a complete access layer for agent data discovery, point lookup, historical query, and unified auth. Production must front it with auth/TLS termination and protect via network isolation and least privilege.

TTL: Separate Management for Different Data Lifecycles

Agent context includes current state, event history, long-term lake data — each with different retention needs. Fluss 1.0 has three configs:

table.log.ttl

and table.log.local-ttl independently manageable with dynamic updates. table.kv.ttl differs: only configurable at primary-key table creation, not dynamically adjustable via ALTER TABLE ... SET.

Row-level TTL expiration is not strict timed deletion; expired records may remain visible, and row-level TTL cleanup does not auto-send delete events downstream. Time-partition retention policies can drop expired partitions independently. Lake-side data is managed by lake retention policies; Fluss TTL does not auto-delete all lake history.

TTL helps control storage and lifecycle but cannot replace compliance deletion, access control, or retention audits.

Server-Side Scan and Point Lookup: Optimized for Interactive Reads, Not General Scan

Fluss 1.0 provides server-side scan for primary-key tables: TabletServer scans current state and returns batches, avoiding client download of snapshot files and log replay. Semantic differences from default snapshot-merge:

Default snapshot-merge merges latest KV snapshot with limited subsequent change logs, recoverable, reflecting a deterministic point in time. server-scan scans server's current KV state directly, no checkpoint recovery; each bucket's scan timestamp determined by scanner open time, so not a global consistent snapshot.

When lake snapshots exist, limited-range reads may use Union Read (lake snapshot + Fluss logs), and server-scan strategy may not apply.

Official positioning: suitable for interactive queries on small primary-key tables (e.g., dashboard refresh, agent tool calls), not a general replacement for large-table, high-concurrency batch processing.

Native Python client provides primary-key lookup:

lookuper = features_table.new_lookup().create_lookuper()
features = await lookuper.lookup({"user_id": 42})

Enables Python AI apps to directly fetch continuously updated user features from Flink without JVM bridging. Latency, failure recovery, connection pooling, concurrency, and caching must be validated under real load.

Predicate Pushdown: Using Column Statistics to Skip Non-Matching Batches

Fluss 1.0 predicate pushdown leverages column statistics (min, max, null count) in log batches to skip batches that cannot satisfy filters. Config table.statistics.columns specifies columns for statistics collection. Prerequisites:

Applies to Arrow-format log tables.

Only effective for batches written after statistics enabled; existing data not auto-rewritten.

Enabling column statistics uses extended log batch format; all Fluss services and clients reading the table must upgrade for compatibility.

Flink SQL applies remaining row-level filters; native Java, Rust, Python, C++ clients must handle residual filtering themselves.

Main effect: reduce unnecessary reads, network transfer, and deserialization via batch-level statistics , not automatic full row-level filtering.

Lakestream: Real-Time and Historical Lake Collaboration, Not Automatic Strong Consistency Everywhere

Lakestream is Fluss's key implementation for lakehouse data collaboration. Fluss provides real-time log and primary-key state access; independent Tiering Service continuously tiers configured data to lake tables. With compatible engines and connectors, Union Read combines real-time layer with historical lake data for real-time context queries and historical analysis.

Union Read does not guarantee all engines, table formats, and access interfaces automatically get a unified real-time view. Fluss 1.0 extends Spark with batch Union Read and time-window incremental reads; actual use must verify compatibility with specific table format, connector, and engine version.

Fluss 1.0 also extends Hudi integration: primary-key tables map to Hudi Merge-On-Read, log tables to Copy-On-Write. For Paimon, historical partition access allows expired Fluss partitions retained by Paimon to continue supporting primary-key point lookup, and log/primary-key tables can write to historical partitions per rules. Limitations: only for Paimon-backed auto-partitioned tables with a single partition column; historical point lookup for primary-key tables, historical writes for log and primary-key tables; cannot combine with Bucket Rescaling. Enabling/disabling requires restarting dependent write and lookup jobs to reload config.

Accurate positioning: Lakestream lowers access and maintenance cost between real-time stream storage and lake storage via collaboration, not a promise that all reads/writes have identical transactional and freshness semantics.

Core Workflow: How Agents Safely Query, Analyze, and Write Back

Production design must separate agent tool interfaces, real-time access, historical query, and business write-back.

Agent receives business goal, determines needed tools — no unlimited access to all tables.

Tool proxy handles authN, parameter validation, authorization, sets timeouts, concurrency limits, call budgets.

Real-time context via dedicated service or native client; historical data via supported compute engines and lake query paths.

Agent combines data, generates result, further validates business rules. High-risk ops (funds, permissions, risk) retain approval and final control by business systems.

Write-back uses idempotency keys, field allowlists, result validation, audit logs. Write success does not guarantee immediate lake-side visibility; visibility depends on commit and tiering progress.

Common mistake: treating Fluss Gateway as complete agent data access layer. Gateway supports metadata, table management, batch writes, but cannot cover missing primary-key point lookup, log scan, etc. Real-time reads, historical SQL, and HTTP writes must be designed separately.

Real Business Scenarios and Engineering Practices

Scenario 1: Agent-Powered Customer Service and Business Assistant

Assistant answers: "What is user's current state? What happened last week? What info explains current issue?" Current profile and recent state kept in primary-key tables suited for low-latency lookup via dedicated data service; historical behavior read via lake tables and compatible engines. Engineering must distinguish current state ("what is now") from historical events ("what happened"). Do not dump all behavior history undifferentiated into real-time state tables. Implement field-level permissions, masking, freshness, and rate limits to prevent agent repeated calls from expanding sensitive data exposure.

Scenario 2: Real-Time Risk Control and Recommendation Decision Support

Real-time risk and recommendation may need latest features, but not all core decisions should be made synchronously by LLMs. For latency-, determinism-, audit-sensitive decision chains, use verified feature services, rule engines, or dedicated decision systems. Agents can participate in anomaly explanation, investigation, suggestion generation, and authorized assisted decisions. If real-time feature access needed, validate lookup latency, cache behavior, timeout fallback, missing data, late-event handling.

Scenario 3: Agent Write-Back of Analysis Conclusions or Business State Updates

Distinguish "generate suggestion" from "execute action". Recommended write-back flow: parameter validation → field allowlist → business rule check → idempotency control → human approval (if needed) → controlled write → audit. Gateway can serve as controlled HTTP write entry but must not be exposed directly to untrusted agent environments. Production entry must solve authN, TLS, network isolation, least privilege. For high-risk write-back, prefer calling business APIs with explicit semantics over allowing agents to arbitrarily modify underlying tables.

Scenario 4: Multi-Engine and Multi-Modal Data Collaboration

Enterprises may need structured analytics, full-text search, vector search, RAG. Unified catalog reduces metadata fragmentation but does not imply single physical copy or identical access paths across engines. Example: lake tables store long-term assets, OLAP serves interactive analysis, search/vector engines serve online retrieval. They must align data sources, schema, update times, permission policies, but storage structures and performance optimizations can differ. Alibaba Cloud OpenLake exemplifies organizing multiple data types under unified metadata catalog with multiple compute/retrieval components serving AI apps — a specific platform implementation, not a mandatory product combination for all.

Performance, Cost, and Technology Selection: Evaluating Agentic Lakehouse

Agent workloads may show multi-turn calls, burst concurrency, repeated requests, unpredictable query shapes — but not all agent requests are higher frequency than traditional BI. Architectural trade-offs must be based on actual workload.

Selection principles:

Real-time context needed: evaluate Fluss, existing feature services, or other current-state access services.

Long-term history and multi-engine access needed: choose open table formats and catalogs meeting engine compatibility.

Natural language analysis needed: prioritize building semantic views, metric definitions, governed SQL tools — not exposing raw tables to agents.

MCP needed: treat MCP as interface layer option while retaining true authN, query authorization, operation audit.

Strict online SLA: decide via real-load testing on query cache, materialized results, OLAP serving, or real-time state service — not default direct lake scans.

Note: sub-second point lookup, minute-level lake freshness, end-to-end agent response latency are three distinct metrics; set targets separately, do not substitute one for another.

Production Issues and Troubleshooting

Issue 1: High Agent Real-Time Lookup Latency

Possible causes: mismatched query strategy, high request concurrency, client connection issues, remote historical access. First distinguish Lookup vs Server-Side Scan. Server-Side Scan targets interactive small-table scans, not large-table batch replacement. For true primary-key Lookup, examine client time, server load, network, cache, access path; adjust based on load-test results.

Issue 2: Predicate Pushdown Yields No Benefit

Check: table uses Arrow log format? Statistics enabled? Filter column type supported? Data read written after statistics enabled? Ensure related Fluss services and clients upgraded to compatible versions (enabling table.statistics.columns involves extended log batch format).

Issue 3: TTL Not Deleting Data as Expected

Do not treat row-level TTL as precise scheduled deletion. Verify table.kv.ttl set at table creation, processing-time vs event-time semantics as expected, compaction cleaned expired records. Separately check table.log.ttl, table.log.local-ttl, partition retention. Lake-side data managed by lake's own retention mechanisms.

Issue 4: Gateway Unauthorized Access Risk

Fluss Gateway 1.0 Preview lacks HTTP caller auth and end-user identity propagation. Must place behind auth/TLS-protected entry with network isolation. Backend service identity must use least privilege; shared account does not justify granting admin rights to all callers. Audit must record real caller and corresponding request, not just indistinguishable shared service identity.

Issue 5: Historical Partition Point Lookup or Write Anomalies

Check table meets Paimon-backed auto-partition, single partition column conditions; confirm historical partition access enabled; check conflict with Bucket Rescaling. Per Fluss 1.0 docs, toggling historical partition access requires restarting dependent lookup and write jobs to reload config.

Issue 6: Agent Answers Correctly but Business Metrics Wrong

May be semantic layer gaps, not query performance. Example: agent used wrong metric granularity, double-counted aggregated data, misunderstood field business meaning. Check semantic views, metric definitions, data lineage, schema docs; validate generated SQL and final results against test sets with known answers.

Troubleshooting principle: Data freshness, query performance, business semantics, security governance must be diagnosed separately. Cannot judge underlying data platform correctness solely from agent's natural language answers.

Advanced Interview and Architecture Reflections

Q1: Fundamental difference between Agentic Lakehouse and traditional Lakehouse?

Not whether LLM integrated, but whether business semantics, discoverable machine interfaces, governed query paths, and automation-suited performance/ops mechanisms are built for agents.

Q2: Why not just give agent a database account?

Database accounts express only partial DB permissions; cannot automatically express each agent's business responsibilities, allowed tools, action scope, full call context. Production needs unified design of authN, authZ, semantics, audit.

Q3: Does Catalog equal unified governance platform?

No. Catalog provides metadata discovery/management; actual permission execution may involve catalog, query engine, storage access control, other policy systems. Capabilities depend on implementation; cannot assume all catalogs support same row policies, column masking, credential management.

Q4: Can MCP directly connect any data source securely?

No. MCP standardizes model-tool interaction; data access scope and permissions controlled by tool services, query engines, identity systems. MCP server's exposed tools and their operations must be explicitly defined and limited by implementer.

Q5: Is Fluss mandatory for Agentic Lakehouse?

No. Fluss suits scenarios needing real-time stream storage, low-latency state access, log consumption, real-time/lake collaboration. If existing OLAP, lake tables, APIs already meet agent freshness and cost needs, adding Fluss may not yield sufficient benefit.

Q6: Does Lakestream mean real-time and historical data have fully consistent semantics?

Cannot infer that. Union Read gives supported engines ability to combine real-time and historical lake data; visibility, consistency, transaction semantics still depend on storage mode, commit method, connector, engine implementation.

Q7: How to measure Agentic Lakehouse maturity?

Not by number of components or agent frameworks integrated. Better metrics: data discovery success rate, semantic layer coverage, SQL result correctness, query latency, privilege escalation test results, audit coverage, write-back idempotency, overall cost per successful task.

Summary, Key Takeaways, and Next Article

This article establishes cognitive framework for AI-agent-oriented lakehouse architecture, but several concepts must be clearly distinguished:

Agentic Lakehouse — architectural direction reorganizing data services, business semantics, query, governance for agents; four-layer architecture is a reference model for engineering design, not a community standard.

Object Storage & Open Table Formats — long-term persistence foundation; Iceberg is important option, not the only choice.

Catalog & Governed Query Layer — connects metadata, identity, access policies, query execution; specific governance capabilities determined by actual product combination.

Semantic Layer & Agent Interface — enables machine users to understand business definitions and call constrained data capabilities; MCP is interface protocol, not governance system.

Apache Fluss & Lakestream — can provide real-time data foundation for scenarios needing latest state, stream consumption, historical lake collaboration; but Gateway 1.0 Preview is not a complete agent query and auth service.

One-sentence conclusion: The essence of Agentic Lakehouse is evolving lakehouse from a human-query data platform into a data infrastructure serving both humans and machines, with business semantics, controlled access, and auditable operations.

Next article (Part 26) continues real-time data foundation research:

"Fluss Real-Time Data Foundation for AI — Primary Key Tables, Delta Join, and 'Latest State' Supply"

Focus: deep dive into primary-key tables, streaming joins, state updates, data lifecycle management, feature serving; further discussion on balancing data freshness, query cost, and system complexity in real-time inference and agent applications.

References

Dremio Agentic Lakehouse Architecture: https://www.dremio.com/blog/agentic-lakehouse-the-architecture-built-for-ai-native-analytics/

Apache Fluss 1.0 Release Notes: https://fluss.apache.org/blog/releases/1.0/

Databricks Unity Catalog Access Control: https://docs.databricks.com/aws/en/data-governance/unity-catalog/access-control/

Fluss Historical Partition Access: https://fluss.apache.org/docs/table-design/data-distribution/partitioning/

OpenLake Official: https://www.alibabacloud.com/help/en/openlake/what-is-openlake

Fluss 1.0 Lakestream & Historical Partition Access: https://fluss.apache.org/blog/releases/1.0/

Fluss Column Statistics & Filter Pushdown: https://fluss.apache.org/blog/releases/1.0/

Fluss Server-Side Scan: https://fluss.apache.org/blog/releases/1.0/

Fluss Gateway Capabilities & Security Limits: https://fluss.apache.org/docs/next/gateway/

HTTP Events into Real-Time Lakehouse: https://fluss.apache.org/docs/quickstart/gateway-lakehouse/

Fluss Gateway Official Docs: https://fluss.apache.org/docs/next/gateway/

Fluss Connector Options: https://fluss.apache.org/docs/engine-flink/options/

Fluss Table Overview: https://fluss.apache.org/docs/table-design/overview/

Fluss 1.0 Python Client Docs: https://fluss.apache.org/blog/releases/1.0/