How Xunlei Boosted Data Processing with Alibaba Cloud EMR Serverless Spark

Background Introduction

Xunlei (NASDAQ: XNET) is a pioneer in distributed technology, building commercial services on a high‑efficiency, trustworthy storage and transmission network. Since its founding in 2003, it has deepened P2P transmission, edge computing, and blockchain, creating a global data network that serves billions of users and underpins Web 3.0 infrastructure.

The company’s technical foundation rests on three capabilities: massive real‑time data governance (processing petabytes of logs per second), a dynamic node scheduling system with millisecond‑level response, and a cross‑scenario federated computing architecture that safeguards privacy while unlocking data value.

Core Business Pain Points

Data processing efficiency bottleneck: the original Hadoop cluster could not fully leverage native acceleration or Remote Shuffle Service, limiting performance and cost‑effectiveness.

Insufficient compute elasticity: fixed resources caused shortages during spikes and idle capacity during off‑peak periods, with long scaling cycles.

High operational complexity: manual resource management, complex Spark upgrades, Python environment handling, and low‑version clusters increased risk and made scaling difficult.

Cost control pressure: workload patterns showed night‑time peaks and day‑time idle, leading to wasted resources.

Technical Upgrade Core Demands

Cost reduction & efficiency: improve data processing speed while lowering operational and hardware costs.

Extreme elasticity: achieve on‑demand, second‑level scaling to match traffic fluctuations.

Simplified operations: eliminate cluster management burden so teams can focus on core development.

Stability & reliability: ensure high‑concurrency processing with fault‑tolerant, resumable jobs.

Alibaba Cloud EMR Serverless Spark Empowerment

The serverless mode removes capacity planning and resource‑pool maintenance from users. Resources are automatically launched for a job, scaled on demand, and reclaimed when idle, providing true elastic scaling and strong isolation.

In the original YARN cluster, memory usage stayed near the upper bound (yarn_cluster_totalMB) with little buffer, causing queuing and congestion when spikes occurred. After migration, the workspace memory consumption curve shows a tidal pattern: memory quickly rises to tens of terabytes during peaks and drops near zero after jobs finish, eliminating the rigid resource pool.

Flexible Access to Archived Data

OSS archive data, stored to reduce storage cost, cannot be read directly and must be thawed. EMR Serverless Spark offers two thawing methods:

Automatic thaw: detects archive files during the planning phase and submits thaw requests automatically. This works for static partitions but may miss dynamic partition scenarios.

Manual thaw: provides a RESTORE TABLE SQL syntax for explicit unfreeze of tables or partitions.

-- Enable automatic OSS archive restore
--conf spark.sql.emr.autoRestoreOssArchive.enabled=true

-- Thaw an entire table
RESTORE TABLE table_name;

-- Thaw specific partitions
RESTORE TABLE table_name PARTITION (pt1='a', pt2='b');

These features let analysts quickly access historical data while keeping storage costs low.

Interactive Development with Kyuubi

Serverless Spark includes a 100 % compatible open‑source Kyuubi Gateway, enhanced for cloud‑native stability and multi‑tenant isolation. It reuses Driver/Executor resources for sub‑second query latency and leverages Spark’s dynamic scaling to release idle resources, delivering cost‑effective interactive analytics.

Xunlei’s data platform integrates Kyuubi with Beeline and Hue, supporting daily warehouse tasks and ad‑hoc queries, dramatically improving development efficiency and reducing analysis costs.

Business and Technical Value Breakthrough

After migration, total cost of ownership dropped sharply because resources are only paid for during actual usage, eliminating idle capacity costs. Managed services improved stability and scheduling efficiency, reducing queuing, retries, and resource contention.

Delivery certainty also improved: large jobs finished about an hour faster, and key reports consistently produced before 06:00, cutting night‑time manual interventions and lowering failure rates.

Future Outlook

Future plans include expanding Serverless Spark to temporary queries, data integration, and other scenarios to further exploit its elastic, zero‑ops advantages. Technically, Xunlei aims to explore AI‑big‑data convergence, leveraging Serverless Spark for massive data processing and AI collaboration.