Optimizing HDFS Storage with Heterogeneous Media, Erasure Coding, and Smart Storage Management

As Hadoop-based big‑data platforms become widespread, many enterprises rely on HDFS as the core storage service. Originally designed for offline batch processing, HDFS focuses on high throughput for large files, but modern workloads introduce new demands: ever‑increasing data volumes, a proliferation of small files, and the need to treat hot and cold data differently.

1. Background

Rapid business growth and the rise of AI/ML increase the amount of data stored in HDFS, driving higher expansion costs. Small‑file handling remains problematic because each file’s metadata resides in the NameNode’s memory, limiting the total number of files to roughly 150 million per NameNode. Additionally, data access patterns vary widely; recent data is accessed far more frequently than older data, making a single storage policy inefficient.

2. Existing HDFS Optimization Techniques

HDFS has evolved to address these issues through heterogeneous storage and erasure‑coding.

Heterogeneous Storage

Since Hadoop 2.6.0, HDFS supports storing replicas on different media types (HDD, SSD, RAM‑DISK, ARCHIVE). Pre‑defined storage classes include:

ARCHIVE – high‑density, low‑power media such as tape, suitable for cold data.

DISK – traditional magnetic disks.

SSD – solid‑state drives for higher performance.

RAM_DISK – in‑memory storage with asynchronous disk backup.

Corresponding storage policies let administrators choose combinations such as:

Lazy_persist – one replica in RAM_DISK, others on disk.

ALL_SSD – all replicas on SSD.

One_SSD – one replica on SSD, others on disk.

Hot – default, all replicas on disk.

Warm – one replica on disk, others on ARCHIVE.

Cold – all replicas on ARCHIVE.

These policies improve resource utilization by placing frequently accessed data on faster media while moving rarely accessed data to cheaper storage. However, they require prior knowledge of each directory’s data temperature, which is often unavailable.

Erasure Coding

Traditional HDFS uses three‑replica redundancy, consuming ~200 % extra storage. Hadoop 3.0 introduced block‑level Reed‑Solomon erasure coding (e.g., RS(3,2), RS(6,3), RS(10,4)), reducing redundancy to as low as 67 % for cold data at the cost of additional CPU for encoding/decoding.

3. Smart Storage Management (SSM)

SSM (Smart Storage Management) automates tiered storage decisions. It collects HDFS metadata and read/write access statistics, infers data hotness, and applies predefined rules to select appropriate storage policies. SSM is an open‑source project led by Intel with contributions from China Mobile; the source code is available at https://github.com/Intel-bigdata/SSM.

SSM follows a Server‑Agent‑Client architecture:

Server – core logic, includes StatesManager (fetches HDFS metadata, tracks hotness) and RuleManager (stores and parses user‑defined rules). Hotness data is persisted in TiDB.

Agent – executes actions on the storage cluster.

Client – provides HDFS‑compatible APIs for applications.

A rule consists of:

Target object (e.g., files matching a path pattern).

Trigger (e.g., daily schedule).

Condition (e.g., accessCount(10min) ≥ 3).

Action (e.g., apply One_SSD policy).

Example rule:

file.path matches "/foo/*": accessCount(10min) >= 3 | one-ssd

This moves files under /foo that receive at least three accesses within ten minutes to a mixed SSD/HDD storage layout.

4. Application Scenarios

SSM is useful for:

Cold‑data tiering – automatically demote long‑idle data from SSD to HDD or ARCHIVE, optionally applying erasure coding.

Hot‑data acceleration – promote frequently accessed files to SSD or RAM_DISK.

Small‑file consolidation – merge many small files into larger containers while preserving a mapping for client‑side retrieval (still under development).

These scenarios reduce storage costs, improve I/O performance, and simplify management of heterogeneous HDFS clusters.

5. Q&A Highlights

HDFS can be deployed in pseudo‑distributed mode on a single node, but 3‑5 nodes are recommended for realistic testing.

SSM is not yet used in production at China Mobile’s provincial platforms; it remains in rapid development.

HDFS is a storage layer, while Spark is a compute engine; they complement each other rather than replace one another.