Apache Ozone: Architecture, Design Principles, and Deployment Guide

Background of Ozone

HDFS is widely used for big‑data storage, but its scalability is limited by the NameNode, especially when dealing with many small files or billions of files. To address these issues, the distributed key‑value storage system Ozone was created; it manages both small and large files and behaves more like an object store.

Ozone

Ozone is a scalable distributed object storage system designed for Hadoop. Components such as Spark, Hive, and YARN can run on Ozone without modification. It can be accessed via command line, Java client API, RPC, or REST.

Ozone consists of volumes, buckets, and keys. Volumes (admin‑created) are analogous to accounts; each volume can contain multiple buckets, but buckets cannot contain other buckets—only keys. Keys are the actual objects stored, uniquely named within a bucket, with no size limit.

Question: Since a key is an object without a directory concept, does each small file in an HDFS directory become a separate key, and how can all related keys be read at once (e.g., by Hive for a partition)?

Design Principles

Strongly Consistent

Architectural Simplicity – a simple architecture eases troubleshooting and debugging, even if some scalability is sacrificed; Ozone can store up to 1 billion objects per cluster.

Layered Architecture – separates namespace management from block and node management, allowing independent scaling.

Painless Recovery

Open Source in Apache

Interoperability with Hadoop Ecosystem – Ozone can be used by existing Hadoop applications (Hive, Spark, MapReduce) via the OzoneFS API.

Hadoop Compatible FileSystem API (OzoneFS) enables Hive, Spark, etc., to use Ozone as a storage layer without code changes.

Data Locality – Ozone supports data locality for upper‑layer applications just like HDFS.

Parallel deployment with HDFS – Ozone can be deployed alongside HDFS and share storage disks.

Architecture

Ozone is composed of three main components: Ozone Manager, Storage Container Manager (SCM), and Datanodes. The diagram below shows their relationships.

Ozone Manager (OM)

OM is a server service responsible for Ozone's namespace, recording all volume, bucket, and key operations—similar to HDFS NameNode. The namespace is a collection of volumes (a forest of trees). OM stores metadata (volumes, buckets, keys) and achieves HA by replicating metadata via Ratis.

Storage Container Manager (SCM)

SCM is analogous to HDFS Block Manager; it manages containers (groups of blocks) and provides block‑level services to OM. Containers consist of independent blocks, and SCM works with datanodes to maintain required replication levels.

Example of a client putKey call:

putKey(keyName, data, pipeline type, replication count)

Parameter explanation: keyName is the file name, data is the content, pipeline type defines the replication strategy (Stand‑Alone or Ratis), and replication count is the number of block replicas. Usually the pipeline type and replication count can be omitted.

The workflow: OM receives a putKey request, asks SCM for a pipeline matching the requested attributes, receives a set of datanodes, stores the mapping {BlockID, ContainerName, Pipeline}, and proceeds similarly to HDFS write flow.

SCM also monitors datanode health, manages pools of datanodes, and handles container lifecycle (creation, closure, replication).

SCM internal structure (illustrated in the diagram below):

Key concepts:

Block – actual data block, can have multiple replicas.

Container – logical collection of blocks.

Pipeline – Ratis (multi‑replica) or Stand‑Alone (single‑replica) replication mode.

Pool – a group of datanodes for easier maintenance and scaling.

Node – physical storage location.

Datanodes

If Ozone runs on top of HDFS, the Ozone datanode runs as a plugin inside the HDFS datanode; otherwise it runs as a standalone daemon. DataNode stores data in containers.

Ozone Client

The client module includes a CLI, a REST handler (supporting RPC and RESTful communication), and a performance testing tool called Freon.

OzoneFileSystem

Ozone implements a Hadoop‑compatible FileSystem interface (OzoneFileSystem) so that applications can access Ozone objects without code changes.

Hadoop Distributed Data Store (HDDS)

HDDS consists of Containers, Ratis, and SCM; it provides a distributed block storage layer without a global namespace. Three datanodes form a Ratis replica chain, each capable of opening multiple containers.

Deployment and Testing

Ozone requires Hadoop 3.0. After installing Hadoop 3.0, download the Ozone package and copy its files into Hadoop’s home directory:

# Execute in Ozone home directory
cp libexec/ozone-config.sh /opt/soft/hadoop/libexec
cp -r share/ozone /opt/soft/hadoop/share
cp -r share/hadoop/ozoneplugin /opt/soft/hadoop/share/hadoop/

Generate Ozone configuration files: ozone genconf etc/hadoop Copy the generated ozone-site.xml to Hadoop’s conf directory and adjust settings as needed.

Add Ozone JARs to the classpath via ~/.hadooprc:

vim ~/.hadooprc
HADOOP_CLASSPATH=/opt/soft/hadoop/share/hadoop/yarn/*.jar:/opt/soft/hadoop/share/hadoop/tools/*.jar:/opt/soft/hadoop/share/hadoop/ozoneplugin/*.jar:/opt/soft/hadoop/share/hadoop/ozone/*.jar:/opt/soft/hadoop/share/hadoop/mapreduce/*.jar:/opt/soft/hadoop/share/hadoop/hdfs/*.jar:/opt/soft/hadoop/share/hadoop/common/*.jar:/opt/soft/hadoop/share/hadoop/client/*.jar:/opt/soft/hadoop/share/hadoop/yarn/lib/*.jar:/opt/soft/hadoop/share/hadoop/tools/lib/*.jar:/opt/soft/hadoop/share/hadoop/ozoneplugin/lib/*.jar:/opt/soft/hadoop/share/hadoop/ozone/lib/*.jar:/opt/soft/hadoop/share/hadoop/mapreduce/lib/*.jar:/opt/soft/hadoop/share/hadoop/hdfs/lib/*.jar:/opt/soft/hadoop/share/hadoop/common/lib/*.jar:/opt/soft/hadoop/share/hadoop/client/lib/*.jar

If running Ozone on top of HDFS, add the following to hdfs-site.xml:

<property>
   <name>dfs.datanode.plugins</name>
   <value>org.apache.hadoop.ozone.HddsDatanodeService</value>
</property>

Start the services in order (NameNode → DataNode → SCM → OM). SCM must be initialized before starting, and OM also requires initialization on first run:

ozone scm --init
ozone --daemon start scm
ozone om --init
ozone --daemon start om

After the services are up, the OM UI is available at http://omserver:9874/.

Basic Ozone commands:

ozone sh volume create --user=work /hive-ozone
ozone sh volume list --user work
ozone sh bucket create /hive-ozone/bucket-test
ozone sh key put /hive-ozone/bucket-test/hadoop.log logs/hadoop.log
ozone fs -put logs/hadoop.log o3fs://bucket-test.hive-ozone/t.log

References

https://cwiki.apache.org/confluence/display/HADOOP/Building+Ozone https://hortonworks.com/blog/introducing-apache-hadoop-ozone-object-store-apache-hadoop/ https://hortonworks.com/blog/apache-hadoop-ozone-object-store-overview/ https://hadoop.apache.org/ozone/docs/0.3.0-alpha/index.html https://hortonworks.com/blog/apache-hadoop-ozone-object-store-architecture/ https://blog.csdn.net/Androidlushangderen/article/details/78168479