This article describes how JD built a multi‑regional, ten‑thousand‑node Hadoop ecosystem, unified resource management with YARN, introduced a three‑level Router scheduling layer, optimized performance, and integrated deep‑learning frameworks to achieve high availability, cost efficiency, and scalable big‑data processing.
This article examines the principles, features, and implementation details of distributed scheduling for big‑data ETL pipelines, covering decentralised schedulers, host selection strategies, fault‑tolerance, operator abstraction, elasticity, trigger mechanisms, visual monitoring, alarm handling, data fan‑in/fan‑out, parameter consistency, real‑time quality checks, lineage tracking, and field‑level traceability.
Juice is an open‑source Mesos‑based distributed task scheduling framework that abstracts Docker tasks, provides REST APIs, leverages Redis queues, and implements custom resource allocation, offering high resource utilization and fault‑tolerant execution for audio‑video processing workloads.
Elastic-Job is a Java-based, decentralized distributed job scheduling framework that addresses limitations of existing solutions by providing features such as distributed coordination via Zookeeper, parallel task execution, elastic scaling, centralized management, customizable workflow tasks, and robust non‑functional requirements, with future plans for multi‑language support and enhanced monitoring.
This article introduces Elastic-Job, a Java‑based distributed job scheduling framework from Dangdang, covering its origins, core features such as sharding and elastic scaling, deployment with Zookeeper, best‑practice usage, open‑source development philosophy, future improvements, and detailed Q&A.
This article summarizes a China Mobile cloud computing talk covering the company’s vision for private and public clouds, core technologies such as lightweight virtualization, distributed scheduling and coordination, the evolution from “small cloud” to “big cloud”, practical reflections on implementation, and strategic recommendations for future development.
