How OpenSDS Gelato Enables Unified Multi‑Cloud Object Storage and Data Migration

Overview

OpenSDS Gelato is an open‑source multi‑cloud data management platform that provides a unified S3‑compatible API for object storage across public clouds (AWS S3, Azure Blob, Huawei OBS) and on‑premises backends (Ceph, FusionStorage). Source code: https://github.com/opensds/Multi-Cloud

Multi‑cloud Object Resource Pooling

Gelato abstracts storage backends as Backend objects. Users create logical Bucket containers via the S3 API; each bucket can be associated with one or more backends, and objects stored in a bucket may reside on different backends. The relationship is many‑to‑many, enabling seamless migration and lifecycle operations while preserving a stable URL.

Backend : physical storage repository (e.g., an AWS bucket, Azure container, Ceph pool).

Bucket : logical container created through the unified S3 API.

Object : data item that belongs to a single bucket.

Backend registration is handled by the Backend Service, which validates metadata and persists it. The S3 Service exposes the unified API and uses adapters to route bucket and object operations to the appropriate backend.

Object Data Cross‑Cloud Flow

Gelato implements policy‑driven data movement between clouds and on‑premises environments. The migration workflow consists of:

Dataflow Manager: user creates a migration plan, which is stored in the database.

Scheduler: reads the plan, generates migration jobs, and publishes them to Kafka.

Datamover: consumes jobs from Kafka, performs the actual transfer using HTTP requests (future versions will integrate third‑party tools), and updates job status.

User can query job progress through the Dataflow Manager.

Planned Enhancements

Future releases (e.g., the June 2019 version) aim to add:

Cross‑cloud lifecycle management: bucket‑level policies for automatic deletion or migration of objects based on age or other criteria.

Integration of external data‑movement tools to support richer backup, replication, and migration scenarios.

Automatic orchestration of data‑flow tasks with cloud‑native applications, such as transferring data for big‑data analytics and retrieving results.