How Alluxio Manages Massive Metadata: Inode, Block, MountTable, and Worker Insights

Introduction

In the era of data-intensive technologies such as IoT, 5G, AI, and autonomous driving, massive and diverse data drives the evolution of distributed storage and processing. Alluxio, an open‑source unified virtual file system, provides a high‑performance layer for accessing various underlying storage systems.

Common Metadata Types in Alluxio

Alluxio Master manages four primary metadata categories: file (inode) metadata, block metadata, mount table, and worker metadata.

File (inode) Metadata

Each file or directory is represented by an inode that stores attributes, permissions, timestamps, and references to data blocks. Inodes may not exist in the underlying storage when using certain write modes or object storage.

Alluxio maintains an InodeTree to represent the hierarchical relationship between files and directories.

It provides a complete set of file‑system operation interfaces with concurrency safety and persistence guarantees.

Journal logging ensures atomicity and durability of every inode operation.

Fine‑grained locking at the inode level enables concurrent read‑write access.

Block Metadata

Blocks belong to files; each file may have zero or many blocks. Block metadata is simpler because blocks lack hierarchical relationships. Alluxio stores block information as two key‑value maps:

<BlockID, BlockMetadata> – records block length.

<BlockID, List<BlockLocation>> – records the worker nodes and storage locations where the block resides.

BlockMetadata is immutable, while BlockLocation lists evolve as blocks are cached or evicted.

MountTable

MountTable manages all mount points, handling creation, updates, and the mapping between Alluxio paths and underlying storage paths.

Worker Metadata

Alluxio Master tracks active workers, their cache contents, and resource usage, including address, start time, space consumption, cached BlockIDs, and pending evictions.

Metadata Storage Modes

2.1 HEAP Mode

In HEAP mode, all metadata resides as Java objects in the JVM heap. Each file consumes roughly 2‑4 KB of heap memory, leading to substantial memory pressure (e.g., 100 million files require 200‑400 GB). GC overhead becomes a critical bottleneck.

2.2 ROCKS Mode

ROCKS mode moves metadata out of the JVM heap into an embedded RocksDB instance, reducing heap pressure. Configuration:

alluxio.master.metastore=ROCKS
alluxio.master.metastore.dir=${alluxio.work.dir}/metastore

RocksDB stores separate directories for inode and block metadata.

2.3 Disk and Memory Usage in ROCKS Mode

Metadata is cached in memory before being flushed to RocksDB. The cache size is controlled by alluxio.master.metastore.inode.cache.max.size. Approximately 100 million files occupy ~4 GB on disk; if the cache limit is not reached, disk usage remains near zero.

2.4 Cache Acceleration and Tuning

When memory is sufficient, increasing the cache size improves performance. However, RPC handling and internal management also consume memory, so users must reserve headroom. Cache eviction follows high/low water‑mark ratios:

alluxio.master.metastore.inode.cache.high.water.mark.ratio=0.85
alluxio.master.metastore.inode.cache.low.water.mark.ratio=0.8

When usage exceeds 85 % of the max cache size, entries are evicted to RocksDB until usage drops below 80 %.

2.5 Switching Between HEAP and ROCKS

The journal format differs between modes, so switching requires restoring from a backup rather than a simple configuration change.

Conclusion

This article used Alluxio 2.8 as an example to reveal the fundamental metadata types and management mechanisms in distributed file systems, and offered practical guidance for optimizing metadata storage and performance.