From a Personal Journey to Data Platform Architecture: Insights on Big Data, Cloud Computing, and System Design

My name is Ai Jia, I have 14 years of work experience and 30 years of programming experience. I work in the Intelligent Platform Department, responsible for tag platforms, tag‑based user segmentation, EasyData, and algorithmic data‑flow architecture, aiming to generalize batch, streaming, and interactive computing and lower the barrier to big‑data processing.

01 Fun

I grew up in the 1980s, started programming on a Subor "learning" computer at age 6, wrote my first BASIC program in 1994, and later designed my first web page in 1998 using Dreamweaver, Fireworks, and Flash.

During university I joined the student tech department, built the college website, experimented with PHP, installed Red Hat Linux and MySQL, and opened the site to the public in 2002.

After graduation I entered a software institute, passed a JavaScript interview by explaining Ajax theory, and later contributed to a 30 k‑line JavaScript project before the language had a module system.

In 2010 I used C++ to solve partial‑differential equations for real‑time dynamics simulation, witnessing the evolution of the C++ standard from C++0x to C++11.

02 Become Yourself

Psychologist Wu Zhi‑hong says the meaning of life is to become yourself; this resonated with my own professional growth.

True Architect

Three years after joining, I delivered a solution for Japan’s largest bank, upgrading a command‑line transaction system to a modern web‑based UI and designing a plug‑in framework that kept business logic independent of specific modules.

Initially I merely relayed the chief architect’s decisions, but I later began independently questioning each design choice, documenting doubts, and finally defending those decisions to clients, earning the chief architect’s respect.

Business‑Agnostic

On a project for an energy company, I built a configurable system that allowed multiple business modules to be integrated without custom code, dramatically reducing cost and increasing revenue.

Big Data Entry

When tasked with an employee reimbursement system handling over a million MySQL rows, I investigated sharding strategies, realized the need for distributed processing, and began studying Hadoop, Spark, HBase, and Storm, eventually using Spark to replace Java workers and halve computation cycles.

We later migrated to cloud infrastructure, leveraging an Alibaba Cloud discount card to scale storage and compute, moving from on‑premise Hadoop/HBase clusters to cloud services.

Cloud Computing Is Forced

At a startup we managed 1 billion mobile‑MAC‑address tags stored in HBase; as data grew to 2 billion we expanded hardware manually, then replaced Java workers with Spark, and finally migrated the whole stack to the cloud to lower operational costs.

Interactive Computing and OLAP

For large‑scale e‑commerce scenarios we needed online, interactive computation; we moved from offline Hive/Spark jobs to OLAP engines (ClickHouse/Doris) and introduced a "bottom‑pool" concept—narrow, scenario‑specific tables—to keep query latency low.

Overall vs. Reductionist Thinking

We contrasted "holistic" data‑flow thinking (treating datasets as whole entities, as in Spark/Flink) with "reductionist" thinking (processing each request individually). To lower the barrier for non‑engineers we proposed a visual data‑flow designer that generates executable pipelines without writing code.

Architect as Building Designer

A software architect is analogous to a building architect: both must understand structural principles, load‑bearing components, and design before construction, while also challenging assumptions and testing edge cases.

Values and Ethics

We discussed the philosophical triad of truth, goodness, and beauty, noting that machines excel at factual truth but lack the capacity for moral or aesthetic judgment, underscoring the need for human‑defined rules to prevent misuse.

Conclusion

The author reflects on a career driven by curiosity, continuous learning, and the desire to create value for both company and society, inviting readers to connect via email ([email protected]) or join the technical community.