Tencent Music Cloud‑Native One‑Stop Machine Learning Platform: Features and Future Roadmap

The article presents a comprehensive overview of Tencent Music's cloud‑native machine learning platform, Cube‑Studio, which aims to simplify end‑to‑end ML workflows by providing one‑stop engineering, distributed acceleration, inference closed‑loop, and edge computing features.

Challenges of traditional ML pipelines are enumerated, such as manual resource requests, fragmented storage, code‑machine binding, and difficulty in hand‑over, which lead to inefficiencies and high operational overhead.

Platform core capabilities include:

Compute orchestration: standardized Linux kernel, upgraded container bandwidth (20 Gb/s), large‑core CPUs, heterogeneous GPUs, and private Docker registry support.

Multi‑cluster deployment: unified UI for project‑level resource pools, mixing public and private resources across clusters.

Distributed storage: unified /mnt/username path, isolated per‑user data, group‑shared storage, and support for high‑performance SSD‑Ceph or CFS.

Online development: integrated Jupyter and VSCode notebooks, customizable images, and Dockerfile abstraction.

Pipeline orchestration: visual drag‑and‑drop editor with templates for TensorFlow, PyTorch, MXNet, Kaldi, Volcano, Ray, Spark, and NNI.

Template development: registration workflow for custom images and parameters.

Debugging: task‑level logs, resource usage view, and Kubeflow‑based flow debugging.

Distributed acceleration addresses framework selection, storage I/O bottlenecks, network latency, and kernel bugs; optimizations include SSD‑Ceph migration, high‑bandwidth networking, and kernel upgrades (Linux 4.14+).

Resource utilization strategies cover CPU multi‑process/coroutine scaling, GPU single‑card utilization improvement, shared‑GPU configurations, and dynamic worker adjustments based on bottleneck analysis.

Scheduler enhancements introduce gang scheduling via kube‑batch, affinity‑aware placement for CPU‑intensive vs. GPU‑intensive tasks, and balanced multi‑project resource pools.

Data skew mitigation categorizes tasks into stateless, ordered‑stateless, and role‑ordered, offering load‑balancing and queue‑based solutions for uneven data distribution.

Inference closed‑loop integrates real‑time data pipelines, model service layers with service mesh, TensorRT acceleration, and supports sparse‑embedding large models via KV storage and dynamic parameter updates.

Edge computing extends the platform to edge clusters, enabling notebook, pipeline, and service deployment at the edge to reduce bandwidth and compute costs.

Code snippet example for NNI hyper‑parameter reporting:

# Report intermediate result
nni.report_intermediate_result(test_acc)
# Report final result
nni.report_final_result(test_acc)
# Define argument
parser.add_argument('--batch_size', type=int)

The article concludes with a summary of optimization directions across code, data, physical layer, and resource management, emphasizing the platform's role in accelerating AI development within Tencent Music.