Data Lake Storage Architecture Selection and JindoFS on Alibaba Cloud

Zheng Kai, a senior technical expert at Alibaba and Hadoop PMC, introduces the data lake concept, emphasizing unified storage of all enterprise data—including structured, semi‑structured, and multimedia—enabling BI and AI analytics.

Key advantages of a data lake are breaking data silos, supporting diverse compute workloads, providing elasticity for cost‑effective scaling, and offering centralized management.

The architecture faces major challenges: massive data volumes (PB‑EB scale), extremely large or deep directory structures, high storage costs, and the need to separate storage from compute while still delivering high‑performance access for varied workloads such as batch, interactive, real‑time, and AI training.

A practical checklist for an ideal data lake storage and acceleration solution includes: (1) object‑storage‑based massive capacity, (2) efficient large‑directory metadata operations, (3) flexible cache‑acceleration, (4) tight integration with compute, (5) support for modern table formats (Delta, Hudi, Iceberg), (6) archiving/compression/security features, (7) comprehensive big‑data + AI ecosystem compatibility, and (8) robust or seamless migration capabilities.

Alibaba Cloud’s JindoFS addresses these points. It builds on OSS with an optimized Hadoop/Spark/AI SDK, improves metadata handling (especially large‑directory rename), and enhances I/O performance. It provides a distributed cache system ensuring metadata and data consistency, write‑through and read‑through caching, load balancing, and LRU eviction.

JindoFS also offers an OSS‑based storage extension with in‑memory metadata caching, fine‑grained locking, data chunking, and performance that surpasses HDFS for both metadata and read paths.

Mapping JindoFS to the checklist shows support for massive object‑storage capacity, superior large‑directory operations, >50% cache acceleration improvement, compute‑aware optimizations via JindoTable, modern table‑format interfaces, archiving/compression/security, full ecosystem compatibility (Hadoop, Flink, TensorFlow, etc.), and partial migration support through an optimized JindoDistCp tool.

Overall, the article provides a comprehensive guide for architects considering a data lake migration, offering both strategic criteria and a concrete solution example.