Design and Implementation of Oceanus Custom HTTP Routing Framework

Oceanus is a unified HTTP service governance framework developed by Meituan's infrastructure team. Built on Nginx and ngx_lua, it provides service registration and discovery, dynamic load balancing, visual management, custom routing, anti‑scraping, session ID reuse, circuit breaking, one‑click traffic cutting, and performance statistics.

The rapid growth of business scenarios at Meituan has made routing increasingly complex, requiring flexible traffic control for scenarios such as:

Group buying flash sales that need multi‑dimensional pressure testing control.

Food delivery traffic isolation by region.

Hotel business gray‑release of a new app version for a tiny fraction of users.

QA environments that forward requests based on custom parameters.

Initially, simple Nginx if directives were used, e.g.:

upstream backend_aaa {<br/>    server 10.4.232.110:8080 weight=10;<br/>}<br/>upstream backend_bbb {<br/>    server 10.4.232.111:8080 weight=10;<br/>    server 10.4.232.112:8080 weight=10;<br/>}<br/>location /abc {<br/>    if($remote_ip = "10.4.242.16") {<br/>        proxy_pass http://backend_aaa; # route to backend_aaa<br/>    }<br/>    proxy_pass http://backend_bbb; # route to backend_bbb<br/>}

While this approach required no extra development and performed close to native Nginx, it suffered from limited condition support, lack of dynamic updates (requiring Nginx reload), and numerous pitfalls when combined with other directives.

To overcome these limitations, Oceanus explored a more powerful solution and investigated existing open‑source projects such as ABTestingGateway, which stores routing policies in Redis and selects upstreams via Lua scripts based on host, IP range, UID range, etc.

Compared with the if method, the AB framework offers dynamic policy updates via HTTP API and richer routing strategies, but it only supports four policy types and still requires custom development for many business needs.

Oceanus therefore designed a highly extensible custom routing framework with four core aspects:

Policy abstraction – a generic structure that can describe a wide range of business scenarios.

Efficient policy query – hierarchical storage and fast lookup.

Runtime policy rendering – template rendering and condition evaluation.

Dynamic group updates – changes take effect without Nginx reload.

Policy Structure Definition

A policy consists of the following fields:

name : human‑readable identifier.

key : the request attribute used for routing (e.g., regionId).

passway : how the key is transmitted (Parameter, Cookie, Header, Body).

condition : an expression template supporting arithmetic, relational, and logical operators.

group : the target upstream cluster.

category : 1 for private (service‑specific) policies, 2 for public policies.

switch : on/off flag.

graylist : list for gray‑release verification.

The design enables any combination of request features (IP, UID, Cookie, etc.) to be matched and routed.

Policy Topology

Private policies are scoped to a specific service (identified by appkey ) and linked to locations via a list of policy names, avoiding data redundancy. Public policies are global and referenced directly by name.

Runtime Retrieval of Location Path

Because Nginx does not expose the original location path during request processing, Oceanus adds two built‑in variables to the core variable array:

{ngx_string("loc_mod"), NULL, ngx_http_variable_loc_mod,<br/>  0, NGX_HTTP_VAR_NOCACHEABLE, 0},<br/>{ngx_string("loc_name"), NULL, ngx_http_variable_loc_name,<br/>  0, NGX_HTTP_VAR_NOCACHEABLE, 0}

These variables store the raw location path and are concatenated with a space to match the format used in the Oceanus management console.

Asynchronous Update Mechanism

Policy data is stored in MySQL. During the init_worker phase, a random worker embeds a timer that periodically pulls the full policy set, parses it, writes to a write‑buffer in shared memory, and then atomically swaps the read buffer so that all workers see the latest policies without reload. A double‑buffer strategy guarantees lock‑free reads.

Policy Query Process

Extract the request Host and matched location_path.

Lookup enabled policy names for that Host+location_path in shared memory.

If the policy is public, fetch its data directly by name.

If the policy is private, use the appkey (derived from the location’s upstream) to locate the specific policy data.

Public policies are prefixed with “oceanus” to distinguish them from private ones.

Runtime Policy Rendering

During the rewrite phase, Oceanus’s rewrite_by_lua_file callback performs the following steps:

Obtain the applicable policies via the query mechanism.

Parse each policy to get its key and passway.

Extract the key’s value from the request according to the passway.

Substitute the value into the policy’s condition template.

Evaluate the condition using Lua’s load function (resulting in true/false).

If true, replace the request’s upstream context with the policy’s group.

Dynamic Group Updates

Group information is stored in ZooKeeper. Changes are propagated via watchers for incremental sync and also via periodic full syncs. Oceanus pushes updates to Nginx memory through local HTTP calls, uses a lock‑free message queue in shared memory, and workers process these messages to update their upstream contexts.

Summary and Outlook

Oceanus’s routing mechanism is already deployed across Meituan’s food delivery, hotel, and in‑store dining services, solving problems such as service set‑based routing, pressure testing, gray releases, and lane environments. Future work includes supporting request‑parameter, header, and cookie modifications, as well as URL‑based dynamic routing.

Author

Zhou Feng, Senior Engineer at Meituan, responsible for key infrastructure projects including Oceanus.