How Inceptor StreamSQL Simplifies Real-Time Data Processing with SQL

Inceptor StreamSQL Overview

Traditional stream processing platforms such as Spark Streaming or Storm require Java or Scala development, raising the entry barrier for data scientists and analysts. Starting with TDH 4.3, Inceptor introduced StreamSQL, allowing users to implement streaming logic using SQL, supporting both simple ETL scenarios and complex PL/SQL features, with event‑driven processing added in version 4.8.

Key Concepts

Stream A Stream represents a data flow and can be an Input Stream (receiving raw data) or a Derived Stream (produced by transforming existing streams).

StreamJob A StreamJob defines the computation on one or more Streams and writes results to a table. Execution is triggered by an Action that starts the associated receivers.

Application An Application groups related StreamJobs, enabling resource sharing and isolation.

Simple StreamSQL Example

Step 1: Create a Kafka data source

Log into a Kafka node, use the scripts under /usr/lib/kafka/bin to create a topic named demo, verify it, and start a producer to publish messages.

Step 2: Create a Stream in Inceptor

Log into Inceptor as the hive user, then execute a StreamSQL statement to create demo_stream that reads from the Kafka topic demo and splits each message into two columns: id (INT) and letter (STRING).

Step 3: Trigger a StreamJob

Create a target table demo_table with the same schema, then insert data from demo_stream into it, which starts the StreamJob.

List running StreamJobs to see their IDs, SQL, and status.

Use the Inceptor UI (default port 4044) to monitor the StreamJob.

After the StreamJob starts, publish messages to the Kafka topic; only messages sent after the job starts are consumed.

Query demo_table to see the ingested records and run further SQL analyses.

Finally, stop the StreamJob with the provided command.

Advantages of StreamSQL

Unified micro‑batch and event‑driven modes Users can switch between processing models within the same system.

High usability Only SQL knowledge is needed to build efficient, stable streaming applications.

Performance gains Special optimizations in StreamSQL can outperform hand‑coded solutions.

Better productization SQL provides a standard interface, simplifying debugging and root‑cause analysis.

Low migration cost Existing SQL logic can be ported to streaming with minimal changes.