End‑to‑End Streaming Data Pipeline with Kafka, Flink, and Apache Griffin

1. Components and Versions

The tutorial uses JDK 1.8, MySQL 5.6, Hadoop 2.7.2, Hive 2.4, Spark 2.4.1, Kafka 0.11, Griffin 0.6.0, and Zookeeper 3.4.1, assuming all are pre‑installed.

2. Kafka Data Generation Scripts

Two Bash scripts are provided: gen-data.sh creates JSON records with dynamic timestamps, random colors, and writes them to /opt/module/data/source.data; streaming-data.sh creates the required Kafka topics and repeatedly runs gen-data.sh every 90 seconds, feeding the data into the source topic.

3. Flink Streaming Processing

The Flink job reads from the source Kafka topic, applies a simple transformation that randomly modifies the name field with a 20% probability, and writes the processed records to the target topic using FlinkKafkaProducer010. The job is defined in flinkProcess.class and uses Gson for JSON parsing.

4. Apache Griffin Configuration and Startup

Two JSON configuration files are shown: dq.json defines the streaming data sources, connectors, pre‑processing steps, and evaluation rules for data quality; env.json configures Spark execution parameters, sinks (Console, HDFS, Elasticsearch), and checkpointing via Zookeeper. The pipeline is submitted with spark-submit to run the Griffin measurement application.

5. Global Code

Two Java classes are included: Student (a POJO with id, name, color, time) and flinkProcess (the main Flink job). The code demonstrates environment setup, Kafka consumer/producer creation, data transformation, and execution.

Note: If malformed data appears in Kafka, the job may error; the article suggests cleaning Kafka data directories and Zookeeper znodes before re‑generating data.