Building and Optimizing Distributed Real‑Time Processing Systems with Machine Learning

Machine Learning Example

A simple linear regression model F(x)=ax+b is trained to predict free‑fall speed from time measurements. Training data (times 1‑6) and test data (times 7‑9) are split. Loss is defined as Loss = |F(x)-Y| / Y. A brute‑force integer search over a∈[-10,10] and b∈[-100,100] evaluates the average loss for each pair; the pair with the smallest average loss (e.g., a=9, b=0.7) is selected if the average loss < 5%.

Distributed Real‑Time Processing Topology

The system processes user‑experience logs in three stages:

Collect Experience Data : Applications write logs to files; an asynchronous agent watches file changes and pushes new entries into a Redis list.

Process Experience Data : A message source reads from Redis, converts rules to an internal format, optionally runs a rule engine, indexes raw JSON into Elasticsearch, and writes aggregated counters to Cassandra.

Store Results : Elasticsearch stores searchable JSON logs; Cassandra stores atomic counters for aggregated statistics.

Reliability is ensured by assigning each tuple a unique 64‑bit ID and tracking acknowledgments with an XOR‑based Acker (similar to Apache Storm). Back‑pressure is handled by a Heron‑style flow‑control where overloaded nodes signal upstream peers to throttle emission.

Message Algorithm Details

Each tuple carries an ack() method. When a node finishes processing a tuple it calls ack(), updating the Acker. The Acker maintains an XOR of all active tuple IDs; when the XOR result becomes zero, all tuples derived from the original spout tuple have been processed. If a tuple is not acked within a timeout, the source retransmits it.

Hurricane Framework

Hurricane is a lightweight Storm‑like engine with four core components:

President : Central scheduler that receives job definitions and assigns tasks to workers.

Manager : Runs on each worker node, receives assignments from President, and forwards tuples to Executors.

Executor : A thread that runs a message loop; for each incoming tuple it invokes its associated Task.

Task : The actual computation, either a spout (source) or a bolt (processor). Each Executor hosts one Task; multiple Tasks can be run on a node.

The architecture supports continuous topologies where streams of tuples flow indefinitely, unlike batch MapReduce jobs.

Future Work

Planned extensions include high‑level abstractions (e.g., a "Squared" API), ordering guarantees, and a BLAS‑based subproject called SewedBLAS for high‑performance linear algebra (integrating MKL/OpenBLAS, CUDA, etc.) to support distributed scientific computing and deep learning.

Source code:

http://github.com/samblg/hurricane