From Early Coding to Big Data Architecture: A Personal Journey Through Data Platforms, Cloud Migration, and System Design

The author, Ai Jia, shares a 30‑year programming background, beginning with BASIC on a Subor computer, progressing through web design, JavaScript development, and C++ scientific computing, ultimately joining JD Retail Intelligence Platform.

Working as an assistant architect, he helped modernize a legacy banking system by creating a plug‑in framework that decouples business logic, enabling reusable modules across multiple financial services.

He later tackled a massive employee reimbursement system, confronting MySQL scaling limits, exploring sharding, and recognizing the need for big‑data technologies such as Hadoop, Spark, HBase, and Storm to simplify data partitioning and aggregation.

At a startup handling billions of device tags stored in HBase, he led hardware upgrades, migrated workloads to Spark, and reduced computation cycles, demonstrating the power of distributed processing.

Facing operational overhead of on‑prem Hadoop clusters, he advocated a cloud migration, leveraging Alibaba Cloud to lower storage and compute costs while improving reliability.

He designed a configurable data‑flow platform that abstracts Spark/Flink pipelines into visual DAGs, allowing non‑engineers to build and deploy data pipelines without writing code, thereby lowering the technical barrier.

The platform supports interactive analytics for large‑scale e‑commerce scenarios, using OLAP technologies (ClickHouse, Doris) and a “bottom‑pool” strategy to partition wide tables by business context.

Throughout, he emphasizes independent thinking, challenging architectural decisions, and the transition from a supporting role to a true architect who can answer client challenges confidently.

Personal reflections interweave themes of self‑realization, the importance of business‑agnostic design, and the continuous pursuit of knowledge across big data, cloud, and system architecture.