Designing a Scalable Data Quality Center for Offline Big‑Data Pipelines

This article describes the design and implementation of a platform‑wide Data Quality Center for offline big‑data pipelines, covering research of existing solutions, design goals, system architecture based on DolphinScheduler, rule definition language, binding and execution mechanisms, and future enhancements such as lineage monitoring and real‑time checks.

Apache GriffinBig DataData Quality

0 likes · 18 min read

Designing a Scalable Data Quality Center for Offline Big‑Data Pipelines

Big Data Technology & Architecture

Mar 16, 2022 · Big Data

End‑to‑End Streaming Data Pipeline with Kafka, Flink, and Apache Griffin

This tutorial demonstrates how to build a complete streaming data pipeline by configuring JDK, MySQL, Hadoop, Hive, Spark, Kafka, and Griffin, generating test data with shell scripts, processing it with Flink, and validating data quality using Apache Griffin in a Spark‑based deployment.

Apache GriffinBig DataData Quality

0 likes · 13 min read

End‑to‑End Streaming Data Pipeline with Kafka, Flink, and Apache Griffin

Big Data Technology & Architecture

Mar 7, 2022 · Big Data

Apache Griffin: An Overview of the Big Data Data‑Quality Monitoring Tool

This article introduces Apache Griffin, a model‑driven big‑data data‑quality monitoring platform, explains its key features, architecture, installation requirements, and provides step‑by‑step usage examples with Hive, Kafka and Spark integration.

Apache GriffinBig DataData Quality

0 likes · 9 min read

Apache Griffin: An Overview of the Big Data Data‑Quality Monitoring Tool