0 views collected around this technical thread.
This article explains how Baidu's vertical search platform tackled massive data growth and scaling challenges by redesigning its data management system, introducing elastic scheduling, decoupling ETCD access, implementing auto‑scaling, and advancing shard expansion to improve performance, stability, and cost efficiency.
The article explains how the vivo Internet Big Data team containerized offline Spark jobs and deployed them with the Spark Operator on a mixed online‑offline Kubernetes cluster, using elastic scheduling and resource‑over‑subscription to boost CPU utilization by 30‑40% and handle over 100,000 daily tasks.
The CNCC2024 forum held on October 24‑26 in Hangzhou showcases industry‑academia‑research integration through keynote speeches, speaker introductions, and detailed abstracts on topics such as AI, privacy computing, graph machine learning, and elastic cloud scheduling, while announcing the CCF‑Ant Research Fund and related initiatives.
This article shares Xiaomi's internal practices of Hadoop YARN, covering scheduling and resource optimization, elastic scheduling, node overcommit handling, federation architecture, metadata warehouse construction, and future plans to improve cluster utilization and cost efficiency.
Shilong Fei from Xiaomi Data Platform presents an in‑depth exploration of elastic scheduling for Hadoop YARN, covering background, design of resource pools, auto‑scaling architecture, challenges such as job stability and user transparency, achieved cost reductions, and future plans for further optimization.
This talk presents Xiaomi's design and deployment of an elastic scheduling system for Hadoop YARN, covering background analysis, resource‑pool strategy, auto‑scaling architecture, stability challenges, label‑based resource isolation, Spark shuffle handling, cost‑saving results and future plans.
ElasticDL is an open‑source distributed deep learning framework built on TensorFlow 2.x and Kubernetes that simplifies programming by letting users define models with the Keras API, while providing elastic scheduling, fault tolerance, and significant performance gains demonstrated through extensive benchmarks.
ElasticDL, the first industry‑level open‑source system for elastic deep learning on TensorFlow, leverages Kubernetes‑native scheduling, fault‑tolerance, and TensorFlow 2.0 Eager Execution to dramatically improve cluster utilization, simplify distributed training, and integrate seamlessly with tools like Kubeflow and SQLFlow.
This article details JD.com's database automation journey, covering their intelligent database management platform (DBS), automated deployment systems, intelligent analysis and diagnostics, and the evolution from traditional containerization to ContainerDB, a smart database service with elastic scheduling and resource optimization.
This article details JD.com's journey from physical servers to a massive Docker‑based cloud‑native platform, covering challenges, architecture, elastic scheduling, monitoring, and resource‑driven operations that support tens of thousands of containers across multiple data centers.