Lakehouse: Mandatory Path or Over-Engineering? A Decision Framework for SMEs

In recent years, Lakehouse has become a major architectural direction in data infrastructure, evolving from traditional Hadoop data lakes and enterprise warehouses to open table formats like Apache Iceberg, Apache Paimon, and Apache Hudi, and now to Apache Fluss exploring real-time stream storage and lake storage collaboration. However, a practical question remains: does Lakehouse becoming an important technical direction mean all enterprises should build one? The answer is negative.

I. Judgment Criteria: Data Scale ≠ Architecture Complexity

Determining whether an enterprise needs a Lakehouse cannot rely solely on data volume or use of technologies like Flink or Spark. More valuable judgment dimensions include:

Data Complexity: Whether data sources, data types, schema changes, and data quality issues are continuously increasing.

Compute Complexity: Whether real-time computing, offline analysis, interactive queries, machine learning, and other diverse workloads coexist.

Data Reuse Degree: Whether the same data must serve multiple teams, systems, and compute engines.

Historical Data Value: Whether long-term retention of raw data, frequent backtracking, or recomputation of historical results is needed.

Real-time Requirements: Whether business needs hour-level, minute-level, second-level, or sub-second data visibility.

Platform Operations Capability: Whether the team possesses data governance, metadata management, failure recovery, performance optimization, and continuous operations skills.

For example, an enterprise with 100 TB of data serving only fixed T+1 reports may be well-served by a mature MPP warehouse. Another with only a few TB but requiring CDC, real-time computing, historical recomputation, multiple analytical engines, and RAG applications simultaneously consuming data may have higher architecture complexity. Data scale determines part of resource pressure; data complexity, access patterns, and business constraints jointly determine architecture complexity.

Lakehouse has formed a relatively mature technical system. Databricks positions Lakehouse as a data management architecture fusing data lake and warehouse capabilities; Apache Fluss further explores real-time stream storage and lake storage collaboration. Vendor definitions of their own architectures do not fully equal product capabilities; implementation must be validated with open-source component versions and enterprise environments. [Source: Databricks, Apache Fluss official documentation]

II. What Problems Does Lakehouse Actually Solve?

Lakehouse is not simply deploying "data lake" and "data warehouse" together. It focuses on providing reliable data management, cross-engine access, and analytical capabilities on an open, scalable storage foundation. [Source: Databricks Lakehouse definition] It primarily solves four categories of problems:

1. Reducing Coupling Between Data and Compute

Traditional warehouses typically organize data storage, compute execution, indexing, and query optimization within a single system — advantageous when analytical loads are stable and query models clear. But when enterprises need more compute engines, data types, and consumption methods, proprietary storage-compute systems may form boundaries.

A typical Lakehouse architecture includes: open table formats organizing table data, snapshots, and metadata; object storage or HDFS persisting data files; Catalog providing table and metadata access entry points for compute engines; Flink, Spark, OLAP, and AI systems handling different processing tasks. This diagram expresses logical relationships — not all engines automatically access all table formats. Actual access depends on engine version, Catalog type, permission configuration, table format features, and deployment environment.

Also, Lakehouse does not mean all compute must read the same physical data, nor that all queries should run directly on object storage. For high-concurrency, low-latency queries, enterprises may still need OLAP internal storage, indexes, caches, or precomputed results.

2. Making Data Assets Easier to Reuse Across Multiple Engines

When data simultaneously serves Flink, Spark, OLAP, and ML tasks, independent data copies per system increase sync pipelines, storage costs, and governance overhead. Lakehouse aims to reduce data exchange costs between compute systems via open table formats and shared metadata.

However, open table formats do not equal complete elimination of data replication. For example, to meet high-concurrency retrieval, online serving, or specific SLAs, enterprises may still import lake data into StarRocks, Elasticsearch, or vector databases. These copies can be reasonable service-layer designs; the key is clarifying authoritative data sources, update mechanisms, timeliness, and consistency requirements.

3. Improving Traceability and Recomputability of Historical Data

Enterprise metric definitions, business rules, and algorithms change. Retaining only final aggregated results makes historical recomputation difficult. Preserving raw data, necessary change records, and manageable historical versions enables:

Historical recomputation after metric definition adjustments;

Backtesting validation after algorithm upgrades;

Downstream rebuilding after data model changes;

Data quality issue localization and repair.

Retaining historical data is only the foundation of recomputability, not the entirety. Enterprises also need data lineage, schema versions, business rule versions, job code versions, permission controls, and reproducible compute environments.

4. Providing Trusted Data Foundation for AI

AI application data needs extend beyond BI reports. Structured data, logs, documents, vectors, business events, and metadata may all become context for retrieval, reasoning, and agent task execution. Lakehouse can undertake part of data storage, organization, and governance but cannot alone replace complete AI data infrastructure.

Lakehouse can provide structured data, historical data, and partial traceability for AI; search engines and vector databases may handle specialized online retrieval services. Metadata, lineage, permissions, and data quality mechanisms help applications understand data sources, applicability scopes, and usage boundaries. Thus Lakehouse can be an important component of AI data infrastructure, but deploying Lakehouse alone does not mean the enterprise has trustworthy AI Context capability.

III. Why SMEs Should Not Blindly Build Full Lakehouse

Lakehouse costs include not just servers, object storage, and compute resources, but engineering complexity from component count. A relatively complete data platform may involve: CDC and messaging systems; Flink, Spark compute engines; Paimon, Iceberg, or Hudi table formats; object storage, Catalog, and metadata management; OLAP query engines; data quality, lineage, permissions, monitoring, alerting; CI/CD, version compatibility, disaster recovery, and failure recovery. These components have dependencies across versions, permissions, metadata, schema, data timeliness, and failure recovery.

For large enterprises with professional platform engineering teams, these investments may yield higher data reuse and lower long-term marginal costs. But for SMEs with only a few data engineers, maintaining Kafka, Flink, Paimon, Spark, StarRocks, and Kubernetes while primarily serving a dozen fixed reports may cause system operations to crowd out business development time. Resulting symptoms: increasing platform components; persistent small-file, compaction, checkpoint, and memory issues; upgrade and compatibility verification consuming massive effort; business metric and data product delivery speed declining. This is typical over-engineering: technical complexity exceeds value current business can generate.

Note: Small teams do not necessarily need to avoid Lakehouse; large data volumes do not necessarily require Lakehouse. The real evaluation is whether platform benefits cover construction, operations, migration, and talent costs.

IV. When Is Traditional OLAP Warehouse Sufficient?

If an enterprise currently exhibits most of these characteristics:

Few data sources, relatively stable business models;

Primarily serving BI, fixed reports, and routine analysis;

Data refresh mainly hourly or T+1;

Clear query patterns, primarily using one analytical engine;

Low historical recomputation frequency;

No obvious data replica synchronization issues;

AI/ML not yet forming stable production data demands.

Then prioritizing a simple, stable, clearly bounded analytics platform is completely reasonable.

Business databases provide data; CDC or ETL handles incremental collection and transformation; OLAP engine handles analytical queries; BI system presents results. This architecture lacks a full Lakehouse but can be a perfectly reasonable production architecture if it meets data timeliness, query performance, reliability, and governance requirements.

Whether to introduce stream processing, open table formats, and multi-engine systems should be driven by real needs. For example: if only hourly refresh is needed, don't forcibly introduce complex stream processing for "real-time"; if only one main analytical engine exists, don't pre-build full Lakehouse for "multi-engine architecture"; if historical data volume and backtracking needs are limited, don't build massive historical data platforms just because object storage is cheap. Architecture design principle: Prioritize solving current problems while avoiding unnecessary obstacles for future evolution.

V. From Lightweight Lakehouse to Real-time Lakehouse: No Need for One-step Arrival

When enterprises start needing long-term raw data retention, historical data sharing, or reducing data sync costs between multiple systems, they can consider introducing lightweight Lakehouse.

A common transition approach: continue using existing OLAP engine for primary queries, while writing data needing long-term retention, recomputation, or cross-engine sharing into open table formats. Object storage handles long-term persistence; compatible compute engines access lake data when needed.

The "open table format" can be Paimon, Iceberg, or Hudi based on actual needs, but should not introduce multiple formats without validation. Format selection must consider write patterns, update methods, snapshot management, engine compatibility, Catalog, operational capability, and existing tech stack.

When enterprises further exhibit needs for second-level data visibility, continuous streaming writes, and unified real-time/historical access, evaluating real-time Lakehouse or Streaming Lakehouse becomes worthwhile.

Taking Fluss as an example, its official architecture explores balance between low-latency stream read/write and long-term historical data access through real-time stream storage and Lakehouse storage collaboration. Its Lakehouse integration involves configuration and independent Tiering Service capabilities — deploying a Fluss cluster alone does not automatically complete all data tiering. [Source: Apache Fluss official documentation]

Fluss handles real-time stream data read/write; Tiering Service continuously organizes and writes data to configured lake storage. Real-time consumers read stream data; external engines supporting corresponding table formats and connection methods can access lake tables.

Three boundaries to note:

Real-time data and lake table data should not be assumed fully visible at the same moment. Actual visibility depends on commit, tiering progress, read methods, and component configurations.

Unified real-time/historical access requires concrete implementation. Whether Union Read is supported, where to read from, and how to handle snapshots and offsets all depend on version and configuration.

Low-latency writes do not mean all queries are low-latency. Query performance still depends on data volume, access patterns, concurrency, merge costs, and execution engine capabilities.

Therefore, real-time Lakehouse is an architectural solution for specific contradictions, not a mandatory next stage for every enterprise after lightweight Lakehouse.

VI. What Signals Indicate Rising Lakehouse Value?

More practical than fixed thresholds by data volume is observing whether the following signals appear in business and platform.

Signal 1: Same Data Needs Repeated Consumption by Multiple Systems

E.g., business data must simultaneously serve Flink, Spark, OLAP, ML, and AI applications. If each system maintains independent data copies, data sync and consistency costs may continuously increase. At this point, evaluate whether open table formats, multi-engine access, and unified metadata can reduce unnecessary replication.

Signal 2: Real-time and Historical Data Begin to Diverge

Real-time systems demand fast writes and reads; historical analysis prioritizes long-term storage, file organization, and scan efficiency. These needs may impose different requirements on data layout and read methods. Lakehouse table formats support historical data management via file organization, snapshots, and metadata; real-time Lakehouse further explores stream storage and lake storage collaboration. But "real-time/historical fusion" doesn't mean all scenarios should introduce Fluss. Enterprises should still evaluate whether existing Kafka, Flink, and lake table pipelines can meet needs at lower total cost.

Signal 3: Historical Recomputation Shifts from Occasional Task to Norm

If business frequently needs recomputing past months or years of data, raw data retention, table format management, historical versions, and reproducible compute environments become more valuable.

Signal 4: Data Starts Serving BI, ML, and AI Simultaneously

Enterprises must then manage not just data files but also metadata, lineage, permissions, data quality, and data usage context. Lakehouse can provide a shared data foundation, but AI retrieval, vector indexing, online serving, etc., usually still need dedicated service components.

Signal 5: Data Replication and Sync Become Persistent Technical Debt

E.g., same business data exists simultaneously in business DB, messaging system, lake tables, OLAP, search engine, and vector database. The first question isn't "whether Lakehouse" but:

Which system is the authoritative data source?

Which replicas are business-essential?

Which data can be read directly?

Which replicas need low-latency updates?

How to verify consistency and timeliness across replicas?

When these issues start affecting delivery efficiency and system reliability, redesigning data architecture may yield significantly higher returns.

VII. Paimon, Iceberg, Fluss, StarRocks: Different Responsibilities, Not Interchangeable

A common tech selection pitfall is comparing components at different layers as if they solve the same problem.

First, Paimon and Iceberg are not simple alternatives. Both belong to open table format ecosystem but differ in design focus, write patterns, and engine support. Validate compatibility with actual business write patterns and target compute engines, not just "real-time" or "open" labels.

Second, StarRocks and Lakehouse are not mutually exclusive. StarRocks can use its internal storage for high-performance analytics, or access supported lake table data via external Catalog. Whether to import data into internal storage should be judged by query performance, concurrency, cost, and data timeliness. StarRocks official documentation lists external data access capabilities for Hive, Iceberg, Hudi, Delta Lake, Paimon, etc., with specific capabilities version-dependent. [Source: StarRocks Catalog official documentation]

Third, Fluss is not an unconditional Kafka replacement. If the enterprise's messaging system already stably meets throughput, persistence, consumption model, ecosystem compatibility, and operational requirements, don't migrate solely because Fluss targets real-time Lakehouse. Validate benefits through actual load, client support, data semantics, and failure recovery testing.

Fourth, Catalog is not the table format itself. Table format describes how to organize and manage table data; Catalog handles table metadata discovery and access. Adopting open table formats doesn't mean different engines automatically share data without configuration.

VIII. How Should Enterprises Formulate Phased Evolution Roadmap?

More reasonable architecture planning isn't setting a fixed upgrade path for all enterprises, but establishing stages that can be entered, paused, or adjusted on demand.

Stage 1: Basic Analytics Platform. Prioritize data quality, query performance, reliability, and business delivery.

Stage 2: OLAP + Open Lake Tables. When historical data accumulation, cross-engine access, or recomputation needs appear, add open data foundation.

Stage 3: Mature Lakehouse. When multi-engine sharing and governance costs rise, strengthen Catalog, lineage, permissions, schema management, and data quality.

Stage 4: Evaluate Streaming Lakehouse. When real-time writes, historical storage, and unified access become prominent contradictions, then evaluate Fluss and similar solutions.

Stage 5: Extend AI Data Infrastructure. When AI applications enter production, build retrieval, vector indexing, context management, permissions, and data lineage capabilities as needed.

This roadmap is not a mandatory maturity ladder. Enterprises can stop at any stage, or skip stages due to existing technical capabilities and business constraints. Evolution should be decided by verifiable returns.

IX. Five Most Common Mistakes in Building Lakehouse

Mistake 1: Building to Prove Technical Advancement

Deploying multiple table formats, compute engines, and storage systems without clear business problems may only increase platform complexity. Improvement: Define target problems and success metrics first — e.g., data sync cost, historical recomputation time, query performance, data freshness, or cross-engine reuse rate.

Mistake 2: Introducing All Components at Once

Kafka, Fluss, Flink, Paimon, Iceberg, Spark, StarRocks, Milvus, Kubernetes each have valid scenarios but are not a mandatory Lakehouse checklist. Improvement: Introduce components by clear architectural boundaries, verifying each solves a problem existing systems cannot economically solve.

Mistake 3: Treating Lakehouse as OLAP Replacement

Lakehouse emphasizes open data organization, persistence, metadata, and multi-engine access; OLAP engines emphasize query execution, aggregation, joins, concurrency control, and low-latency analytics. Improvement: Decide based on query SLA whether data should compute directly on lake or enter dedicated OLAP serving layer.

Mistake 4: Only Calculating Storage Cost, Ignoring TCO

Low object storage unit price doesn't mean overall system is cheap. Data reads, network traffic, compute, caching, metadata management, compaction, failure recovery, and labor all generate costs. Improvement: Use full-lifecycle Total Cost of Ownership (TCO) evaluation, including construction, migration, operation, scaling, and exit costs.

Mistake 5: Building Lakehouse Without Clear Data Authority Boundaries

If lake tables, messaging systems, OLAP, search engines, and vector databases each maintain data copies without clear authoritative sources, sync mechanisms, and quality validation, Lakehouse core benefits are hard to realize. Improvement: Define authoritative data sources, derived data purposes, sync timeliness, retention policies, rebuild methods, and consistency verification rules.

X. Final Verdict: Lakehouse Is Not a Mandatory Path; Architecture Evolution Capability Is

Returning to the initial question: Is building Lakehouse a technical frontier or over-engineering? The correct answer is not binary.

For businesses with simple operations primarily serving fixed BI, mature OLAP warehouse may be the most suitable solution.

For enterprises beginning to see historical data accumulation, recomputation, and multi-engine sharing needs, lightweight Lakehouse may be a cost-controllable evolution path.

For enterprises with continuously growing real-time data scale, prominent real-time/historical access contradictions, and increasing maintenance costs of existing data pipelines, real-time Lakehouse deserves serious evaluation.

For large enterprises with multiple teams, multiple data domains, simultaneously serving BI, ML, and AI applications, open table formats, metadata governance, real-time data infrastructure, and dedicated serving layers may jointly constitute long-term data platform capabilities.

Three principles must always be upheld:

Business problems precede technical components. Clarify what problem to solve before deciding whether to introduce new technology.

Overall cost precedes single-point metrics. Cannot only look at storage cost, single-query performance, or a component's throughput.

Evolution capability precedes one-time stack. Need not pre-build all capabilities, but must avoid unnecessarily locking critical data, metadata, and business logic into a single system.

In one sentence: Keep architecture simple when business is simple; gradually open data foundation as data complexity rises; evaluate real-time Lakehouse when real-time/historical contradictions become prominent; build AI data infrastructure matching business after AI enters production.

Lakehouse is not a stage all enterprises must experience. The truly indispensable capability is the enterprise's ability to continuously judge its own problems, control architecture complexity, and smoothly evolve with business growth. This is what mature data architecture design should pursue.