Zero‑Downtime Upgrade Strategies for Xianyu Search Service

Background – In the Internet industry, frequent online service upgrades are routine. Over the past quarter, Xianyu engineers performed thousands of releases, updating more than a million lines of code.

Search Service Architecture – Xianyu’s search stack consists of Search Planner, Query Planner, Rank Service, and the Heaven Ask 3 engine, forming a set of independent micro‑services that are orchestrated through the Search Planner. Additional business‑logic and gateway layers sit on top, creating a request chain that spans dozens of clusters and hundreds of servers.

Compatibility Assurance – Before any upgrade, forward and backward compatibility must be verified. Guidelines include making RPC calls tolerant of unknown or missing parameters, deprecating rather than removing fields, distinguishing default from absent values, and creating new interfaces when necessary.

Stateless Service Upgrade – For services designed without state (e.g., Java micro‑services, Search Planner), the upgrade process is straightforward: determine batch size based on minimum availability, stop a batch of containers, update images, wait for the batch to become healthy, then proceed to the next batch.

Stateful Service Upgrade – Stateful components require more careful handling. Common approaches are: (1) hot‑update at the gateway layer (e.g., Nginx), (2) progressive rollout where new requests are routed to the new version while the old instance drains, and (3) full‑copy deployment with dual‑write synchronization before switching traffic.

In the case of the Heaven Ask 3 engine, the chosen method involves creating a brand‑new engine instance, full data sync, incremental sync, gradual traffic shift, and finally decommissioning the old engine.

Service Discovery – A robust service‑discovery mechanism provides distributed consistency, graceful service registration/deregistration, load balancing, traffic control, and cross‑region failover, acting as the “valve” for traffic during upgrades.

Risk Control – Following Alibaba’s three‑pronged safety principle: (1) comprehensive monitoring of key metrics, (2) mandatory gray‑release before full rollout, and (3) ready‑to‑use rollback procedures that can be executed within seconds.

Summary – Successful zero‑downtime upgrades rely on service decoupling, compatibility checks, ordering based on dependencies, stateless vs. stateful upgrade paths, and pre‑planned monitoring, gray‑release, and rollback strategies.