Athena Data Factory: A One‑Stop Data Development and Governance Platform – Architecture, Features, and Impact

Background – Spark Thinking, an online education leader, faced explosive data growth and limitations in data access, reliability, and ETL efficiency, prompting the need for a unified, high‑performance data development and governance solution.

Product Overview – Athena Data Factory provides a full‑stack platform covering offline development, real‑time development, operations, self‑service data extraction, data mapping, and console management, enabling users from operations, BA, product, R&D, and finance to develop, schedule, monitor, and consume data with minimal friction.

Key Modules – The platform includes an offline development module (HiveSQL, DorisSQL, Python, data sync), a real‑time development module (Spark, Flink, Flink SQL), an operations center (dashboards, task rerun, lineage), a self‑service extraction module (Hive, Spark, Presto APIs), a data‑map module (catalog, lineage, metadata), and a console module (project, source, permission, queue, sensitive‑data management).

Architecture – Four‑layer design: Interaction layer (Vue + Monaco), Service layer (micro‑services, API, permission, resource management), Engine layer (Hive, Spark, Presto, Airflow, multi‑tenant support), and Component layer (Airflow, Tencent Cloud SCF, etc.), ensuring scalability, extensibility, and cloud‑native deployment.

Use Cases & Scenarios – Supports T/H+x batch processing, minute‑level analytics, real‑time streaming (Spark Streaming, Flink, Iceberg), and AI pipelines (data preprocessing, model training, inference), covering a wide range of business needs.

Implementation & Operations – Launched in late 2021, iteratively added Doris, SparkSQL, API generation, real‑time module, and AI support; migrated Hadoop to EMR, introduced versioned GitLab integration, and refined monitoring, alerting, and cost‑control mechanisms.

Results & Benefits – Daily execution of ~20,000 offline tasks, 470 internal users (130 MAU), 92 % of new tasks from non‑big‑data roles, 2,700+ tasks created in 2023, 30 %+ reduction in task runtime, 29 % cost reduction, and significant productivity gains (3 + hours saved per task).

Future Outlook – Plans include AI‑assisted debugging, enhanced observability, containerised core services, and further cloud‑native optimisation to sustain growth and innovation.