Real‑Time KMeans Streaming with Sophon & Slipstream: From Model Training to Kafka Prediction

As data volume and richness grow, enterprises increasingly rely on machine learning (ML) to extract value, especially for advertising recommendation and business forecasting where low‑latency, continuous data streams demand real‑time analytics.

Basic Idea

Use Transwarp Sophon’s Midas client to train a KMeans model on sample data and export the model as a JSON file.

Import the JSON model into Transwarp Slipstream, create a streaming job, and apply the model to incoming real‑time data.

Persist prediction results to an Inceptor or Hyperbase table, or forward them to Kafka for immediate visualization.

Training the Model with Sophon

In Sophon, drag the data source module onto the workflow canvas, configure the sample dataset, add the KMeans operator, and finally use the "Write JSON File" operator to export the trained model to /tmp/kmeans.json on HDFS.

Streaming Prediction with Slipstream

Create a Kafka topic named unlabeled to receive data from a producer:

./kafka-topics.sh --create --zookeeper tw-node3228:2181 --topic unlabeled --partitions 1 --replication-factor 1

Define the input stream and result table in Slipstream:

create stream unlabeled(id int, c1 double, c2 double) tblproperties("topic"="unlabeled","kafka.zookeeper"="172.16.3.228:2181","kafka.broker.list"="172.16.3.230:9092");
create table kmeans_predict(id int, c1 double, c2 double, predict int);

Run the inference query that loads the exported model and predicts clusters based on fields c1 and c2:

insert into kmeans_predict select *, kmeans_predict(c1, c2, cast("/tmp/kmeans.json" as string)) from unlabeled;

The built‑in kmeans_predict(col1, col2, ..., model_path) function takes feature columns and the path to the JSON model.

Start a Kafka producer (e.g., kafka-console-producer) and send sample records:

bin/kafka-console-producer.sh --broker-list tw-node3230:9092 --topic unlabeled
6,2.2,3.3
7,5.4,4.5
8,10.2,3.4
9,4.2,6.8
10,2.4,9.7

Result Verification

Query the kmeans_predict table to view clustering outcomes:

select * from kmeans_predict;

Conclusion

Sophon supports many common algorithms beyond KMeans, and when combined with Slipstream it enables a wide range of streaming machine‑learning applications. Real‑time ML eliminates the need to wait for batch data, allowing immediate insights for advertising, industrial control, traffic management, and other domains that require instant predictions.