Open Source Distributed Object Storage Solutions Overview

Object Storage Service (OSS) provides massive storage for binary files such as images, documents, audio, and video. Apart from public‑cloud offerings, private‑cloud environments often consider open‑source distributed object storage solutions, which are listed below for reference.

Concept Overview

Block Storage

Typically SAN (Storage Area Network) products, such as hard drives and disk arrays, belong to block storage.

File Storage

NAS (Network Attached Storage) products like CephFS, as well as systems such as GFS and HDFS, are examples of file storage.

Object Storage

Combines the high‑speed direct‑disk access of SAN with the distributed sharing characteristics of NAS, usually accessed via RESTful APIs.

Open Source Solutions

Swift

Swift is a core OpenStack project, an elastic, highly available distributed object storage system implemented in Python under the Apache 2.0 license. It provides a RESTful HTTP Object Storage API for creating, modifying, and retrieving objects and metadata.

Overall, enterprises seeking scalable distributed object storage clusters can consider Swift.

Ceph

Ceph is a high‑performance, highly available, scalable distributed storage system that provides unified object, block, and file storage, implemented in C/C++.

Its object storage supports two interfaces:

1. S3‑compatible: a large subset of the S3 RESTful API.

2. Swift‑compatible: a large subset of the OpenStack Swift API.

Ceph is an enterprise‑grade distributed storage system suitable for building object storage services and private cloud platforms.

MinIO

MinIO is an enterprise‑grade, S3‑compatible object storage system written in Go under the Apache 2.0 license. It supports multiple client languages (Java, Python, Go) and offers high concurrency, making it suitable for storing massive images, videos, and documents.

For big‑data integration, MinIO works with Spark, Presto, Hive, Flink, and supports formats such as Parquet, JSON, CSV, including compression and encoding.

MinIO is designed primarily for AI/ML workloads but is also suitable for other big‑data scenarios, making it a strong open‑source object storage choice.

HBase MOB

HBase MOB leverages Apache HBase 2.0’s Medium Object Storage feature to store binary data (100 KB–10 MB) such as images, documents, audio, and short videos. MOB data is written to special MOB files stored in dedicated regions, similar to HBase + HDFS architecture.

Supported in Apache HBase 2.0, CDH 5.4.x, and HDP 2.5.x and later, allowing users to build custom object storage services on top of HBase.

Hadoop Ozone

Ozone is an Apache Hadoop sub‑project that provides distributed, scalable object storage to address HDFS’s limitations with small files. It builds on the Hadoop Distributed Data Store (HDDS) and integrates with Spark, Hive, and YARN. It is currently in alpha and not recommended for production.

Conclusion

Object storage solves the challenge of storing massive images, documents, and media. The heavyweight solutions are Swift and Ceph, each with distinct strengths; HBase MOB is notable within the Hadoop ecosystem, while lightweight options like MinIO are also viable. MongoDB’s GridFS offers another alternative. Choose based on actual requirements.

References:

Ceph: The De Facto Standard for Object Storage

Evaluating Two Major OpenStack Object Storage Technologies: Swift vs. Ceph

Why Object Storage Is Gaining Traction Worldwide?