Why Kappa Beats Lambda: A Deep Dive into Modern Big Data Architectures

Preface

Following the October 10 article "Deep Dive into Big Data Lambda Architecture", we further explore the integration of batch and real‑time processing within a single system, focusing on the limitations of Lambda and the advantages of the newer Kappa architecture.

Lambda Architecture Review

The core idea of Lambda is to split a big‑data system into three layers: Batch Layer, Speed Layer, and Serving Layer. The Batch Layer stores the full data set and pre‑computes batch views. The Speed Layer processes incremental data to produce real‑time views. The Serving Layer merges batch and real‑time views to answer queries.

Kappa Architecture

Kappa was proposed by Jay Kreps to address Lambda’s need for two separate code bases. It unifies batch and real‑time processing using a single code path built on technologies such as Spark and Spark Streaming.

Use Kafka or a similar distributed queue to retain raw data for as long as needed.

When a full recomputation is required, launch a new stream processing job that reads all retained data and writes results to a new store.

After the new job finishes, stop the old instance and clean up its results.

Comparison of the Two Architectures

Data processing capability : Lambda handles massive historical data; Kappa’s historical processing is more limited.

Machine overhead : Lambda requires both batch and speed layers to run continuously, leading to higher resource consumption; Kappa runs full recomputation only when needed, reducing overhead.

Storage overhead : Lambda stores both batch and real‑time results; Kappa stores only the final results, saving space.

Development and testing difficulty : Lambda needs two code bases, increasing complexity; Kappa uses a single framework, simplifying development.

Operations cost : Maintaining two systems in Lambda raises OPEX; Kappa’s single‑framework approach lowers it.

StreamSQL and Lambda Architecture

Transwarp StreamSQL is a stream‑processing engine that supports SQL and PL/SQL, enabling developers to write a single SQL program for both offline and real‑time workloads. It can ingest data from Kafka, store results in various formats (TEXT, ORC, Holodesk, HBase), and integrate stream data with historical tables for richer analytics.

Kappa Architecture Case Study

We built a smart‑traffic system using StreamSQL as the stream engine. Vehicle license data from Kafka is processed to detect clone‑vehicle behavior. A user‑defined function DetectCloneVehicle(param1, param2) flags two vehicles with the same plate when their straight‑line distance exceeds param1 kilometers within param2 minutes.

Initial parameters (20 km, 2 min) yielded low detection efficiency. Adjusting to (10 km, 5 min) improved performance dramatically. The following SQL jobs illustrate the parameter tuning:

CREATE STREAM vehicle_stream1(license STRING, location STRING, time TIMESTAMP)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
TBLPROPERTIES ("topic"="fakeLicense", "kafka.zookeeper"="172.16.1.128:2181", "timefield"="time", "timeformat"="yyyy-MM-dd HH-mm-ss.SSS");

CREATE TABLE clone_vehicle_result_app1(license STRING, location STRING, time TIMESTAMP);

INSERT INTO clone_vehicle_result_app1
SELECT DetectCloneVehicle(20, 2) AS cloned
FROM vehicle_stream1
HAVING cloned > 0;

CREATE STREAM vehicle_stream2(license STRING, location STRING, time TIMESTAMP)
ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
TBLPROPERTIES ("topic"="fakeLicense", "kafka.zookeeper"="172.16.1.128:2181", "timefield"="time", "timeformat"="yyyy-MM-dd HH-mm-ss.SSS");

CREATE TABLE clone_vehicle_result_app2(license STRING, location STRING, time TIMESTAMP);

INSERT INTO clone_vehicle_result_app2
SELECT DetectCloneVehicle(10, 5) AS cloned
FROM vehicle_stream2
HAVING cloned > 0;

Conclusion

Lambda and Kappa are two prevalent big‑data system designs aimed at unifying batch and real‑time computation. While Lambda offers strong historical processing, its dual‑code‑base approach incurs higher development and operational costs. Kappa simplifies the stack by using a single framework, reducing overhead and easing maintenance. However, for scenarios requiring distinct batch and real‑time models (e.g., separate machine‑learning pipelines), Lambda may still be appropriate. StreamSQL extends Kappa with a unified SQL interface, HA guarantees, and flexible storage options, making it a compelling choice for building modern real‑time analytics platforms.