How NetEase’s Object Storage Powers Scalable Media and Data Services

Speaker Introduction

Wang Tao, senior cloud storage product manager at NetEase Hive, shares the design and application scenarios of NetEase Object Storage (NOS).

1. Object Storage Application Scenarios

Unstructured data such as images, videos, files, and backups require a scalable cloud storage platform. NOS provides a one‑stop solution via a comprehensive SDK, supporting typical scenarios:

Resource distribution and download: integrates NOS with NetEase CDN for video, image, and app package delivery.

UGC services: edge‑node upload acceleration enables mobile clients to upload data without exposing NOS keys.

Enterprise drive: secure file sharing and collaboration.

Cloud data processing: easy image and video processing after upload.

Website/app static‑dynamic separation: static assets served via BGP or CDN to reduce web server load.

2. Core Competitive Features of NOS

2.1 Distributed Database (DDB)

DDB is a distributed relational database developed by NetEase Hangyan Backend Technology Center, used for metadata storage. It addresses massive structured data storage and high‑concurrency read/write access.

2.2 Distributed File System (DFS)

DFS (Distributed File System) handles massive unstructured data with high‑concurrency throughput, complementing DDB for metadata.

2.3 Rich Media Services

Since 2013, NOS has provided image processing services for products like Lofter, NetEase Cloud Music, and others. The service evolved from a few servers handling a few hundred TPS to dozens of servers handling over 20,000 TPS, supporting cropping, watermarking, chain processing, and more.

Key components of the image processing pipeline (named Tobie) include:

Interface Layer : HTTP Restful service (e.g.,

http://nos.netease.com/doc/Koala.jpg?imageView&thumbnail=100x0

) built on libevent C++ server.

Concurrency Layer : Workers fetch tasks from a queue; Processor integrates GraphicsMagick for images and ffmpeg for video, with Lua extensions for custom commands.

Data Interface Layer : Provides in‑memory and POSIX file system access via nosfs (FUSE‑based), converting HTTP Restful calls to file operations.

2.4 Overload Control

To prevent cascading failures in latency‑sensitive services, NOS implements queue length limits, request time‑outs, and health‑check‑driven load shedding. Requests exceeding thresholds are rejected, and overloaded nodes are bypassed using Nginx health checks.

2.5 Architecture Highlights

Core services are isolated: upload/download, image processing, and metadata handling run on separate clusters, reducing the impact of a single node failure.

3. Upload Acceleration (NCAN)

Edge nodes are deployed close to users to split the upload path into two phases: low‑bandwidth/low‑RTT client‑to‑edge and high‑bandwidth/high‑RTT edge‑to‑data‑center. Optimizations include:

Custom upload protocol supporting arbitrary chunk sizes and append semantics.

Mobile upload optimization using TCP pipeline to fully utilize WAN bandwidth.

WAN TCP/HTTP tuning: increasing send buffer size to avoid bottlenecks.

These measures dramatically improve upload success rates and throughput.

4. Q&A Highlights

Lua is used for on‑the‑fly image processing rule adjustments without recompilation.

Go powers many backend services, replacing thread‑based servers like Tomcat for higher concurrency.

Video processing currently offers basic functions (watermark, thumbnail, format conversion); advanced quality detection is not yet implemented.

Storage backend is a custom‑built distributed file system optimized for small files, not Ceph.