How Meituan Scaled Its Pipeline Engine to Power 100k Daily Jobs

Background

Continuous delivery, first introduced at the 2006 Agile conference, has become essential for improving development efficiency. Meituan initially adopted Jenkins to quickly support business needs, but as the number of pipelines grew, the limitations of short‑term, fragmented tooling became apparent.

Problems and Approaches

Business Introduction

A pipeline is a series of automated steps—such as build, code scan, test, and deployment—executed in order to transform code into a deployable product, forming a directed acyclic graph.

Main Challenges

Scheduling efficiency bottleneck : Short‑lived jobs (seconds to minutes) are sensitive to scheduling latency; existing open‑source tools (Jenkins, GitLab CI, Tekton) use monolithic scheduling that can cause queuing and service unavailability under peak load.

Resource allocation : Jobs often outnumber available resources; static provisioning leads to either resource exhaustion or low utilization, especially for time‑critical tasks.

Tool differentiation : Diverse tools require different integration methods (APIs, code snippets, custom runtimes), making it hard to hide tool‑specific details from pipeline authors.

Solution Ideas

Separate scheduling decision from resource allocation to eliminate the scheduling bottleneck and enable horizontal scaling of both modules.

Introduce a resource‑pool management model that tags jobs and resources, allowing flexible matching and dynamic pool resizing.

Design a layered component architecture (system interaction layer, execution resource layer, business logic layer) to standardize interfaces and support plug‑in extensions.

Overall Architecture

The engine consists of a Trigger, Task Center, Decision Service, Workers, and Component SDK. Triggers launch pipelines, the Task Center records job states, Decision Service selects runnable jobs, Workers pull jobs and allocate resources, and the SDK executes component logic.

Core Design Points

Job Scheduling Design

Scheduling is split into decision and resource allocation. Decision Service evaluates job dependencies, priorities, and conditions, then marks jobs as pending . Workers pull pending jobs, execute them, and report results back to the Task Center, which updates the job state and triggers the next decision round.

Resource Pool Design

A multi‑queue system maps job tags to resource pools. Each queue is associated with a single tag, while each pool can serve multiple tags, balancing isolation and utilization. Labels are two‑dimensional (component and pipeline) to enable fine‑grained matching and dynamic scaling.

Component Layered Design

Components implement a minimal set of methods (init(), run(), queryResult(), uploadArtifacts()) defined by a template. The system interaction layer enforces a uniform state machine, while the execution resource layer abstracts different runtimes (Docker, VM, external services). Adapters (e.g., ShellCommand) provide out‑of‑the‑box implementations for common scenarios, reducing developer effort.

Results

After three years of iteration, the engine unifies backend infrastructure for services such as Meituan Store, Delivery, Dianping, and more, supporting Java, C++, NodeJS, and Go. It processes nearly 100,000 pipeline executions per day with a success rate above 99.99%.

Future Plans

Leverage serverless and other cloud‑native technologies to create lighter, more elastic resource management, improving startup latency and isolation.

Build an end‑to‑end component development platform to streamline component creation, deployment, and operation, fostering a vibrant ecosystem of reusable tools.