How Ctrip Handles 20 Billion Daily Requests: Inside Its High‑Performance API Gateway

Overview

Ctrip introduced its API gateway together with the micro‑service architecture in 2014. By July 2021 the gateway handled more than 3,000 services and processed 200 billion requests per day. The initial implementation followed Netflix OSS concepts and a Zuul 1.0‑style code base, using Tomcat NIO + AsyncServlet, an independent thread‑pool business chain, Apache HttpClient for synchronous calls, and core components Archaius, Hystrix and Groovy.

Full Asynchronous Redesign

Async Flow Design

The gateway adopts a fully asynchronous model: server‑side async, business‑process async, and client‑side async, all built on Netty’s NIO/Epoll event‑loop architecture. Typical async scenarios include business I/O (e.g., request validation, authentication) and raw I/O (e.g., reading the first bytes of a message). Challenges identified are flow design & state transition, exception handling (including time‑outs), context propagation, thread scheduling and traffic control.

RxJava provides the reactive backbone. Filters implement a unified contract returning a Maybe<T> that can emit a value, be empty, or signal an error.

public interface Processor<T> {
    ProcessorType getType();
    int getOrder();
    boolean shouldProcess(RequestContext context);
    // Unified external contract
    Maybe<T> process(RequestContext context) throws Exception;
}

public abstract class AbstractProcessor implements Processor {
    protected void processSync(RequestContext context) throws Exception {}
    protected T processSyncAndGetResponse(RequestContext context) throws Exception {
        process(context);
        return null;
    }
    protected Maybe<T> processAsync(RequestContext context) throws Exception {
        T response = processSyncAndGetResponse(context);
        return response == null ? Maybe.empty() : Maybe.just(response);
    }
    @Override
    public Maybe<T> process(RequestContext context) throws Exception {
        Maybe<T> maybe = processAsync(context);
        if (maybe instanceof ScalarCallable) {
            return maybe; // sync shortcut
        } else {
            return maybe.timeout(asyncTimeout(context), TimeUnit.MILLISECONDS,
                Schedulers.from(context.getEventloop()), timeoutFallback(context));
        }
    }
    protected long asyncTimeout(RequestContext context) { return 2000; }
    protected Maybe<T> timeoutFallback(RequestContext context) { return Maybe.empty(); }
}

The engine stitches inbound, outbound, error and log stages using RxUtil.concat to concatenate filter callables. Errors trigger the error stage, after which processing returns to the outbound stage before finally entering the log stage. A global timeout guards the whole pipeline.

public class RxUtil {
    public static <T> Maybe<T> concat(Iterable<? extends Callable&lt;Maybe&lt;T&gt;&gt;> iterable) {
        Iterator<? extends Callable&lt;Maybe&lt;T&gt;&gt;> sources = iterable.iterator();
        while (sources.hasNext()) {
            Maybe<T> maybe;
            try { maybe = sources.next().call(); }
            catch (Exception e) { return Maybe.error(e); }
            if (maybe != null) {
                if (maybe instanceof ScalarCallable) {
                    T response = ((ScalarCallable<T>)maybe).call();
                    if (response != null) return maybe; // short‑circuit
                } else {
                    if (sources.hasNext()) {
                        return new ConcattedMaybe(maybe, sources);
                    } else {
                        return maybe;
                    }
                }
            }
        }
        return Maybe.empty();
    }
}

public class ProcessEngine {
    private void process(RequestContext context) {
        List<Callable<Maybe<Response>>> inboundTask = get(ProcessorType.INBOUND, context);
        List<Callable<Maybe<Void>>> outboundTask = get(ProcessorType.OUTBOUND, context);
        List<Callable<Maybe<Response>>> errorTask = get(ProcessorType.ERROR, context);
        List<Callable<Maybe<Void>>> logTask = get(ProcessorType.LOG, context);
        RxUtil.concat(inboundTask)
            .toSingle()
            .flatMapMaybe(response -> {
                context.setOriginResponse(response);
                return RxUtil.concat(outboundTask);
            })
            .onErrorResumeNext(e -> {
                context.setThrowable(e);
                return RxUtil.concat(errorTask).flatMap(r -> {
                    context.resetResponse(r);
                    return RxUtil.concat(outboundTask);
                });
            })
            .flatMap(r -> RxUtil.concat(logTask))
            .timeout(asyncTimeout.get(), TimeUnit.MILLISECONDS,
                Schedulers.from(context.getEventloop()),
                Maybe.error(new ServerException(500, "Async-Timeout-Processing")))
            .subscribe(unused -> {
                logger.error("this should not happen, " + context);
                context.release();
            }, e -> {
                logger.error("this should not happen, " + context, e);
                context.release();
            }, () -> context.release());
    }
}

Streaming Forward & Single‑Threaded Execution

Only the HTTP start‑line and headers are parsed; the body is not stored. If the body arrives after the header, it is forwarded immediately when the upstream connection is already selected, otherwise it is temporarily buffered until business logic finishes, then sent together with the header.

Earlier entry into the business pipeline lets upstream services receive the request sooner, reducing gateway‑induced latency.

Compressing the body lifecycle reduces memory pressure on the gateway.

Streaming introduces three challenges: thread‑safety when different stages run on different threads, multi‑stage coordination (e.g., a half‑received request whose upstream connection is already established), and edge‑case handling such as premature upstream errors or client disconnects.

To mitigate these, Ctrip runs the entire request flow—Netty server, business logic and Netty client—on the same event‑loop thread, binds the request context to that loop, and isolates resources such as connection pools. This single‑threaded model eliminates concurrency bugs and simplifies error handling, at the cost of limiting worker threads to the number of CPU cores and requiring strict avoidance of blocking I/O inside the loop.

Other Optimizations

Lazy loading of internal variables (e.g., parsing cookies or query strings only when needed).

Off‑heap memory and zero‑copy techniques to further cut memory usage.

Migration to JDK 11 with TLS 1.3 and the Z Garbage Collector; GC pause times dropped dramatically while CPU usage increased modestly.

Custom HTTP codec that handles malformed requests (e.g., request smuggling) and enables traffic‑governance logic to run on otherwise rejected traffic.

Gateway Business Forms

The gateway acts as a unified ingress point, providing three main values:

Decoupling of heterogeneous network environments (intranet vs. internet, different security zones, etc.).

Common cross‑cutting concerns such as security, authentication, routing, gray‑release, rate‑limiting, circuit‑breaking, monitoring and alerting.

Fine‑grained traffic control, including private protocols (SOTP), link optimization and multi‑region active‑active deployment.

Gateway Governance

Multi‑protocol compatibility is achieved by abstracting protocol handling (similar to Tomcat’s HTTP/1.0, 1.1, 2.0 support) and defining a common intermediate model. The gateway supports HTTP, SOTP and other request‑response protocols.

Routing is expressed as JSON objects that describe match type, value, matcher, tags, properties and target endpoints with weights for gray releases. Example:

{
  "type": "uri",
  "value": "/hotel/order",
  "matcherType": "prefix",
  "tags": ["owner_admin","org_framework","appId_123456"],
  "properties": {"core": "true"},
  "routes": [{
    "condition": "true",
    "zone": "PRO",
    "targets": [{"url": "http://test.ctrip.com/hotel", "weight": 100}]
  }]
}

Module orchestration is managed by a control plane that assigns modules (e.g., adding response headers) to specific stages (PRE_RESPONSE, POST_RESPONSE, etc.) with ordering, gray‑ratio, conditions and parameter templates. Example configuration:

{
  "name": "addResponseHeader",
  "stage": "PRE_RESPONSE",
  "ruleOrder": 0,
  "grayRatio": 100,
  "condition": "true",
  "actionParam": {
    "connection": "keep-alive",
    "x-service-call": "${request.func.remoteCost}",
    "Access-Control-Expose-Headers": "x-service-call",
    "x-gate-root-id": "${func.catRootMessageId}"
  },
  "exceptionHandle": "return"
}

Governance also includes unified management of protocol adapters, routing tables and module deployment across independent gateway clusters, reducing operational complexity as the number of business branches grows.

Conclusion

Ctrip’s gateway combines a fully asynchronous Netty core, RxJava‑driven filter pipelines, streaming request handling and a modular governance framework to sustain 200 billion daily requests. The design balances performance, scalability and maintainability, and continues to evolve with new protocols (e.g., HTTP/3) and service‑mesh integration.