How OPPO Cloud Storage Leverages Kubernetes and CSI for Cloud‑Native Solutions

Distributed Storage Overview

Three main types of distributed storage are introduced: object storage (e.g., AWS S3, Alibaba OSS) which manages whole files via HTTP APIs; distributed file systems (e.g., CephFS, GlusterFS) that provide POSIX‑compatible file interfaces; and distributed block storage that exposes raw block devices via SCSI/iSCSI for formatting or block APIs.

OPPO Cloud Platform Storage Architecture

The block storage service consists of three core components: a gateway that parses iSCSI requests and forwards them to storage nodes, a storage layer that handles read/write operations on disks, and a cluster manager that maintains metadata and health arbitration for the cluster.

Cloud‑Native Storage Landscape

Kubernetes has become the de‑facto platform for cloud‑native workloads, and storage solutions such as OpenEBS, Portworx, and Rook integrate with it. Cloud‑native storage not only serves applications but also exploits Kubernetes features—declarative APIs, container isolation, and resource efficiency—to simplify operations and improve reliability.

Kubernetes Storage Abstractions and CSI

Kubernetes abstracts storage lifecycle through four resources:

StorageClass : defines provisioner, parameters, and reclaim policy.

PersistentVolume (PV) : represents a provisioned storage unit (file system, block device, or cloud disk).

PersistentVolumeClaim (PVC) : a request for storage with specific size and access mode.

Container Storage Interface (CSI) : a standardized set of APIs that allow Kubernetes to manage storage creation, attachment, expansion, and snapshotting across heterogeneous back‑ends.

CSI Sidecar Components

To bridge Kubernetes resources with CSI drivers, several sidecar containers are used:

external‑attacher – triggers ControllerPublishVolume.

external‑provisioner – handles provisioning calls.

node‑driver‑registrar – registers the CSI plugin with kubelet.

external‑resizer – enables volume expansion.

external‑snapshotter – provides snapshot functionality.

livenessprobe – maps CSI health checks to Kubernetes liveness probes.

Kubernetes‑CSI Interaction Diagram

PV Scheduling and Node Operations

During scheduling, PV affinity influences pod placement, allowing administrators to control where pods run based on storage locality.

Two critical CSI calls are explained:

NodeStageVolume – mounts the remote cloud disk on the host node.

NodePublishVolume – binds the staged volume into the container’s filesystem.

Storage Containerization

Managing storage as containers involves using StatefulSet to provision a unique PVC for each pod (e.g., ${claimName}-${podName}) and deploying the gateway component via DaemonSet on every node to handle storage requests.

Benefits of Storage Containerization

Reduced operational cost : Kubernetes provides rolling updates, health checks, and auto‑scaling via simple YAML manifests, replacing complex Ansible scripts.

Lower development overhead : StorageClass, PV, and PVC abstractions enable declarative management, easy querying via labels, and seamless integration with Prometheus for monitoring.

Improved isolation : Container runtime limits CPU/memory per pod, preventing resource contention and increasing overall utilization.