How GF Securities Leverages Lambda/Kappa Architectures for Real-Time Stock Analytics

Background

Complex Event Processing (CEP) has been a core technology in the securities industry for over 15 years, supporting algorithmic trading, risk management, order and liquidity analysis, quantitative trading, and signal generation.

CEP is typically combined with time‑series databases to provide a seamless continuum of historical and real‑time data streams.

Lambda Architecture Overview

The original Lambda architecture, introduced by Nathan Marz, separates processing into batch, speed, and serving layers, offering robustness, low‑latency reads, scalability, generalization, extensibility, ad‑hoc queries, minimal maintenance, and debuggability.

batch view = function(all data) realtime view = function(realtime view, new data) query = function(batch view, realtime view)

Its immutable data model simplifies recomputation and fault tolerance but introduces operational complexity when handling ad‑hoc queries over large data volumes.

Kappa Architecture

Jay Kreps proposed Kappa, replacing the batch layer with a single streaming framework (e.g., Kafka + Spark Streaming) to simplify development and operations.

Use Kafka as the durable log, retaining enough history for re‑computation.

When code changes, replay from the required offset.

Switch downstream consumers to the new stream once it catches up.

Retire the old stream.

GF‑Lambda Implementation

GF Securities built a hybrid “GF‑Lambda” architecture that blends Lambda and Kappa concepts to meet financial industry constraints such as T+1 settlement and heavy reliance on Oracle databases.

Processing is divided into three categories:

Spark SQL for batch preprocessing.

Spark Streaming for mini‑batch real‑time processing.

Storm for true low‑latency streaming.

Key advantages include:

Pipeline orchestration with Airbnb’s Airflow (script‑based, dynamic task generation).

Redis caching of service state to avoid costly checkpointing.

Batch layer runs Spark SQL on Oracle data, minimizing database load while achieving near‑real‑time freshness.

Application Scenarios

1) Price and Volatility Alerts

Using MongoDB for user preferences, the platform joins real‑time market data from Kafka with alert settings to generate high‑frequency price and percentage‑change notifications.

2) Real‑Time Stock News

Oracle data is loaded into Spark via JDBC, pre‑processed with Spark SQL, and published to Kafka. Downstream services match news with user subscriptions, leveraging Redis caching and Airflow scheduling.

3) Capital‑Flow Trading Strategy

The strategy computes real‑time capital‑flow indicators from high‑frequency trade data, derives signals based on thresholds (e.g., large‑order inflow > outflow, flow information content > 50%, flow intensity > 20%), and issues buy/sell orders via Kafka‑driven micro‑services.

Additional components include real‑time P&L tracking (batch layer) and automated stop‑loss handling.

Conclusion

The GF‑Lambda platform demonstrates how a finance‑focused big‑data ecosystem can free developers from low‑level infrastructure concerns, enabling rapid creation of valuable analytics such as alerts, news, and trading signals, while adhering to industry‑specific constraints.