Why Disaggregated Storage‑Compute Architecture Is Revolutionizing Big Data Platforms

With rapid advances in hardware, especially network and storage performance, and cloud providers promoting hardware‑software co‑acceleration, many enterprises now build data services and data lakes on cloud storage, requiring a separate compute layer for analytics. This leads to the Disaggregated Storage and Compute Architecture.

Background

Apache Hadoop introduced a "storage‑compute integrated" model where compute tasks run on the same nodes that store the data, minimizing network overhead. In contrast, a disaggregated architecture decouples storage (e.g., HDFS, HCFS, relational databases) from compute engines such as Spark, Presto, or Flink.

The main advantages are:

Easier provision of data‑analysis services for diverse business needs and avoidance of duplicate hot data.

Independent scaling of compute and storage, reducing storage costs as compute demand typically grows faster.

Improved stability through resource isolation between compute and storage services.

Architecture Goals and Technical Requirements

Key goals include flexible data openness, independent scalability of compute and storage, strong resource isolation, and performance comparable to integrated architectures. Achieving parity requires better network/storage hardware, advanced scheduling, or caching strategies.

StarRing Data Platform’s Disaggregated Design

StarRing adopts Kubernetes and Docker, abstracting CPU, GPU, memory, SSD, and HDD into a unified resource pool. The platform consists of four layers: cloud operating system, data storage, compute, and application services.

The cloud OS exposes declarative interfaces for storage volumes, CPUs, GPUs, etc., enabling elastic scaling without code changes. Application services run in containers, supporting micro‑services, .Net, and traditional apps.

Storage services (HDFS, Search, etc.) are containerized and use local volumes for high‑performance data storage. Compute services (database nodes, Spark, TensorFlow) are also containerized, with the scheduler placing compute tasks close to the data nodes based on metadata, achieving near‑integrated performance.

Elastic scaling is realized through two strategies: time‑based scaling for predictable workloads and workload‑based scaling for dynamic demand.

Cloudera’s Approach to Disaggregation

Cloudera also leverages Kubernetes and Docker to improve service isolation and data‑analysis flexibility. Since 2019, Cloudera has been developing a containerized big‑data platform, deploying its Machine Learning product on Kubernetes. However, core storage and compute components (Hive, HDFS, HBase, Kudu, Impala) remain on a private cloud base and are not fully containerized, limiting flexibility.

To enhance data‑lake flexibility, Cloudera introduced Ozone, an S3‑compatible storage layer that redesigns metadata management, supporting billions of files and using Raft for distributed consistency.

Ozone’s interface supports multiple protocols (Hadoop OFS, O3FS, S3) and offers Java APIs and CLI tools, facilitating migration of existing Hadoop applications and new S3‑based workloads.

Conclusion

Disaggregated storage‑compute architectures enable enterprises to build flexible, scalable, and isolated big‑data platforms while maintaining performance comparable to integrated designs. Implementations such as StarRing’s TDC and Cloudera’s Ozone illustrate how containerization, Kubernetes orchestration, and thoughtful resource abstraction drive the evolution of modern data lakes and warehouses.