Build a Lightweight, High‑Availability Real‑Time Stream Processing System

When discussing stream processing, Storm and Spark Streaming are popular but have drawbacks: Storm can amplify failures, and Spark Streaming consumes large memory, leaks, and depends on Hadoop.

For beginners, these issues feel like a black box; source code is extensive. Small to medium enterprises often need simple solutions without complex environments.

Inspired by the open‑source lightweight distributed real‑time computing framework light_drtc , we propose a zero‑knowledge approach to building a lightweight, highly available stream processing system.

We define the end‑to‑end pipeline—from data collection to near‑real‑time computation and final storage—as an information waterfall, divided into three parts: data collection, task coordination management, and task computation.

1. Data Collection (CN): Use message queues such as Kafka or RabbitMQ for real‑time ingestion, leveraging their load‑balancing capabilities. Each collector node uses Zookeeper to watch task manager nodes, distributing mini‑batch streams based on a unique ID hash.

Data collectors can send data to the task coordination layer via efficient RPC frameworks like Thrift or Avro.

2. Task Coordination Management (AN, multi‑master): Upon startup, coordination nodes register with Zookeeper and listen to task computation nodes. They receive real‑time streams, apply simple mini‑batch processing, and dispatch tasks to computation clusters, adjusting assignments dynamically.

During high‑traffic periods, combine memory and disk storage to avoid overloading memory, processing tasks in queue order.

Coordination nodes also use Thrift/Avro to forward data to computation nodes.

3. Task Computation: Computation nodes register with Zookeeper, then process received mini‑batch data using a map/reduce‑like fork/join model. Results are sent back upstream.

Result storage can be any of the popular NoSQL solutions such as MongoDB 3.x, Redis 3.x, AeroSpike 3.7.x, or Elasticsearch 5.x.

This outline provides a practical, lightweight alternative for building real‑time stream processing systems.