Shepherd Unified API Gateway: Architecture, Design, Implementation and Future Roadmap

Background

In micro‑service architectures, service decomposition leads to exponential growth of APIs, making API management a critical challenge. Shepherd is Meituan's internally developed, fully self‑served unified API gateway designed to replace traditional web‑layer gateways and enable developers to expose functionality and data through configuration.

1.1 What is an API Gateway?

An API gateway sits between external requests and internal services, providing protocol conversion, authentication, rate limiting, parameter validation, monitoring, and other common functions for the multitude of APIs generated by micro‑service splits.

1.2 Why build Shepherd?

Before Shepherd, developers had to build custom web applications for each API, handling authentication, rate limiting, logging, and protocol conversion manually, which was inefficient and resource‑intensive. Shepherd centralizes these capabilities, improving developer productivity and resource utilization.

1.3 Benefits of Shepherd

Improved development efficiency : configuration‑driven API publishing, built‑in auth, rate limiting, circuit breaking, and extensible custom components.

Reduced communication cost : automatic generation of API documentation and client SDKs.

Higher resource utilization : serverless‑style full‑managed API hosting eliminates the need for developers to manage machines.

Technical Design and Implementation

2.1 Overall Architecture

Shepherd consists of three main planes:

Control Plane : Shepherd Management Platform and Monitoring Center for lifecycle management and metric collection.

Configuration Center : Uses Meituan's Lion configuration service to distribute routing, rules, and component configurations.

Data Plane : The Shepherd server processes requests, invokes backend RPC/HTTP/function services, and returns responses.

2.1.1 Control Plane

Developers create APIs, input parameters, generate DSL scripts, test via mock, approve releases, monitor failures, and finally decommission unused APIs—all through a configuration‑driven workflow that typically takes less than ten minutes.

2.1.2 Configuration Center

API configurations are expressed in a custom DSL and stored in Lion with local caching, enabling dynamic, zero‑downtime updates.

2.1.3 Data Plane

Requests are routed using two in‑memory structures: a direct MAP for static paths and a prefix‑tree for paths with variables, avoiding costly regex matching.

Functional components such as tracing, monitoring, validation, authentication, rate limiting, circuit breaking, and gray‑release are applied based on DSL configuration.

2.2 High‑Availability Design

Performance Optimization : Fully asynchronous processing with Jetty (later Netty) I/O threads, server‑side request pre‑warming, and async logging raise QPS from 2k to >15k.

Service Isolation : Cluster isolation per business line, fast/slow thread‑pool segregation, and custom thread pools prevent slow APIs from affecting others.

Stability Guarantees : Multi‑dimensional traffic control, request caching, timeout management, and circuit‑breaker mechanisms.

Request Security : Integrated signature verification, SSO, UAC/UPM, Passport, merchant auth, anti‑scraping, etc.

Gray‑Release : Supports API‑level and downstream service gray‑release with flexible strategies.

Monitoring & Alerting : 360° metrics (business, machine, JVM) and comprehensive alerting dashboards.

Self‑Healing : Elastic scaling based on CPU, rapid node/component removal.

2.3 Usability Enhancements

Automatic DSL generation from service JSON schema.

Quick API creation for RPC, HTTP, SSO callback, and Nest APIs.

Batch operations to modify multiple APIs simultaneously.

Import/Export of API configurations across environments.

2.4 Extensibility

Custom components allow developers to plug in bespoke logic (e.g., custom signing, result handling).

Service orchestration via integration with Meituan's Pirate middleware enables multi‑service composition without burdening the client.

Future Roadmap

Shepherd now serves over 18,000 APIs across 90+ clusters with billions of daily calls. The next year focuses on cloud‑native evolution, static site hosting, and a component marketplace.

3.1 Cloud‑Native Evolution

Migrate the gateway to Meituan's Serverless platform Nest, extract core functions into SDKs, and allow developers to include only needed components, reducing WAR size and improving security.

3.2 Static Site Hosting

Provide a unified solution for hosting static resources, managing custom domains, authentication, and CI/CD integration.

3.3 Component Marketplace

Enable developers to share custom components with the broader Meituan engineering community, fostering an ecosystem and avoiding duplicate effort.

Authors: Chongze, Zhiyang, Li Min (Meituan Infrastructure Team)