How to Harness Event‑Driven StreamSQL for Low‑Latency Real‑Time Analytics

Inceptor StreamSQL is a class‑SQL declarative language that runs on the Transwarp Slipstream engine, which combines event‑driven and micro‑batch processing to support both low‑latency and high‑throughput tasks.

Event‑Driven Stream Processing

In event‑driven mode, StreamSQL reads each incoming record, applies business logic, and outputs the result immediately, offering lower latency than micro‑batch (mini‑batch) processing.

Enabling Event‑Driven Mode

Configure the StreamSQL service in Transwarp Manager by setting the NGMR_ENGINE_MODE parameter to morphling. To run in micro‑batch mode, set it to mapred.

Application Example – Emily’s Tasks

Emily needs three low‑latency tasks. Before triggering a StreamJob, she sets streamsql.use.eventmode=true. The following statements illustrate three typical scenarios.

1. Load data from a stream table into a regular table

SET streamsql.use.eventmode=true;

CREATE STREAM s1(score INT, name STRING) TBLPROPERTIES("topic"="tps1","kafka.zookeeper"="tw-node127:2181","kafka.broker.list"="tw-node127:9092");

CREATE TABLE t1(score INT, name STRING);

INSERT INTO t1 SELECT * FROM s1;

When the job runs, the submitted StreamJob appears as a persistent Active Job (see image).

2. Window Stream aggregation into a regular table

StreamSQL supports two window types: sliding windows (defined by LENGTH and SLIDE) and hopping windows (where LENGTH = SLIDE). Windows can be split by system time or event time.

Emily creates a window stream using event time, with LENGTH = 4 seconds and SLIDE = 2 seconds, aggregates the data, and inserts the result into a regular table:

SET streamsql.use.eventmode=true;

SET streamsql.use.eventtime=true;

CREATE STREAM s1(score INT, name STRING, ts STRING) TBLPROPERTIES("topic"="tps1","kafka.zookeeper"="tw-node127:2181","kafka.broker.list"="tw-node127:9092","timefield"="ts","timeformat"="yyyy-MM-dd HH:mm:ss","use.lowlevel.consumer"="true");

CREATE STREAM s1win AS SELECT * FROM s1 STREAMWINDOW (LENGTH '4' SECOND SLIDE '2' SECOND);

CREATE TABLE t1(score INT, name STRING);

INSERT INTO t1 SELECT SUM(score), name FROM s1win GROUP BY name;

3. Join two Window Streams and insert into a regular table

Emily creates two window streams (both LENGTH = 4 s, SLIDE = 2 s), joins them on the name field, and writes the result to a table:

SET streamsql.use.eventmode=true;

SET streamsql.use.eventtime=true;

CREATE STREAM s1(score INT, name STRING, ts STRING) TBLPROPERTIES("topic"="tps1","kafka.zookeeper"="tw-node127:2181","kafka.broker.list"="tw-node127:9092","timefield"="ts","timeformat"="yyyy-MM-dd HH:mm:ss","use.lowlevel.consumer"="true");

CREATE STREAM s1win AS SELECT * FROM s1 STREAMWINDOW (LENGTH '4' SECOND SLIDE '2' SECOND);

CREATE STREAM s2(class INT, name STRING, ts STRING) TBLPROPERTIES("topic"="tps2","kafka.zookeeper"="tw-node127:2181","kafka.broker.list"="tw-node127:9092","timefield"="ts","timeformat"="yyyy-MM-dd HH:mm:ss","use.lowlevel.consumer"="true");

CREATE STREAM s2win AS SELECT * FROM s2 STREAMWINDOW (LENGTH '4' SECOND SLIDE '2' SECOND);

CREATE TABLE t1(score INT, class INT, name STRING);

INSERT INTO t1 SELECT s1win.score, s2win.class, s1win.name FROM s1win JOIN s2win ON s1win.name = s2win.name;

Conclusion

Slipstream’s hybrid support for event‑driven and micro‑batch modes enables users to achieve sub‑5 ms latency for latency‑sensitive applications (e.g., online fraud detection) while still providing high throughput for batch‑oriented workloads (e.g., video traffic analysis). StreamSQL’s mode‑switching semantics let users flexibly leverage Slipstream’s real‑time computing capabilities to meet diverse business requirements.