Design and Implementation of XStore: A Hadoop‑Based Sample Storage System

In the 360 Internet Technology Training Camp (Session 16) the speaker introduced XStore, a Hadoop‑based sample storage system designed to handle massive APK and other binary samples for cloud scanning.

Core security workflow: newly detected samples are uploaded to the storage system, scanned by multiple engines, and results recorded; known samples are directly retrieved.

Storage requirements include real‑time upload, download for detection services, and retrospective scanning when detection models change.

Target sample characteristics: average size 50 MB, total capacity 10‑100 PB, billions of files, primarily write‑once read‑many, random and sequential reads, high availability and backup.

Non‑functional requirements: real‑time access, Hadoop compatibility, horizontal scalability, strong backup and disaster recovery, high sequential read performance, and security.

Technical stack: HDFS for file storage combined with HBase for metadata, using large .mrg files to merge small samples; file headers store offset, length, checksum, and name for random access.

System architecture layers:

Data layer – HDFS, HBase, ZooKeeper providing basic storage interfaces.

Storage Logic layer – file‑header protocol, file merging into fixed‑size blocks, and index service.

Data Access layer – file addressing, redundancy handling, distributed lock via ZooKeeper, and CRUD operations.

Network layer – file chunking, RPC/HTTP APIs, load balancing for stable service delivery.

Service layer – online HTTP/RPC interfaces with PHP/Python SDKs, offline Hadoop jobs for back‑scanning and scheduling.

Since launch XStore stores ~1 billion samples (≈21 PB), handling 20 TB daily throughput, 2‑3 k QPS, and supports both online and offline scanning workloads.

Optimizations include adding a file cache service between front‑end machines and storage to decouple upload latency from backend failures, improving reliability.

For more details, the full PPT and video are available via the 360 technology public account.