Evolution and Performance Optimization of a High‑Throughput HTTP Gateway at Ximalaya

The gateway is a mature middleware used by many internet companies to handle public business features efficiently; at Ximalaya it serves over 200 billion daily calls with peak QPS exceeding 40 k, supporting more than 500 web services for over 600 million users.

Version 1 employed Tomcat NIO with AsyncServlet, providing basic reverse‑proxy functions but suffering from blocking I/O, excessive object caching, memory copies, and connection‑close handling, which limited throughput to around 5 k QPS and caused frequent full GC.

Version 2 switched to Netty with a fully asynchronous, lock‑free, layered architecture. The access layer handles HTTP codec and protocol‑level monitoring; the business‑logic layer implements public features such as user authentication, black‑/white‑listing, flow control, intelligent circuit breaking, gray release, fine‑grained downgrade, traffic scheduling, traffic copy, and request‑log sampling using a responsibility‑chain pattern; the service‑call layer manages asynchronous remote calls with Netty’s connection pool, ensuring non‑blocking operations.

The connection pool reuses HTTP connections efficiently, closing them on conditions like Connection:close, idle timeout, read/write timeout, or FIN/RESET, and avoids premature reuse that could cause 400 errors.

A comprehensive full‑link timeout mechanism covers protocol parsing, queue waiting, connection establishment, waiting for a connection, pre‑write timeout checks, write timeout, and response timeout, ensuring robust failure handling.

Monitoring and alarm systems provide second‑level alerts and metrics, reporting to a management platform that aggregates data into InfluxDB; both protocol‑level (e.g., attack detection, oversized requests) and application‑level (e.g., latency, QPS, bandwidth, error codes, connection stats, failure rates, traffic jitter) metrics are captured.

Performance optimizations include object‑pool techniques to reduce allocation and GC pressure, minimizing thread‑context switches by adjusting asynchronous‑to‑synchronous configurations, GC tuning (large young generation, SurvivorRatio = 2, max tenuring = 15), and careful logging practices to prevent Log4j from blocking Netty I/O threads. The following finalize method illustrates a GC‑related link‑cleanup hook:

/**
 * Cleans up if the user forgets to close it.
 */
protected void finalize() throws IOException {
    close();
}

Future plans involve migrating to HTTP/2 for multiplexed connections, further refining monitoring and alarm accuracy, and enhancing downgrade mechanisms to ensure graceful degradation across the entire site.

In summary, the gateway has become a standard component in the company's infrastructure, and the shared experiences aim to provide practical insights for building and evolving high‑performance, reliable gateway systems.