WeChat NLP Algorithm Microservice Governance: Challenges and Solutions

WeChat's NLP algorithm services are deployed as a large number of micro‑services, even a relatively small feature such as the book‑recommendation function triggers thousands of RPC calls.

The rapid growth of micro‑services introduces three major challenges: (1) management – how to efficiently develop, test, and deploy many algorithm services; (2) performance – how to keep inference latency low; (3) scheduling – how to achieve dynamic load‑balancing among many identical services.

To address the management challenge we built an automated CI/CD pipeline that packages a Python function into a ready‑to‑run micro‑service using predefined templates, and we introduced task‑aware auto‑scaling that expands or shrinks instances based on queue backlog.

For service composition we model the top‑level workflow as a DAG, each node representing a micro‑service call. A domain‑specific language (DSL) and web‑based visual tools allow engineers to construct, stress‑test and deploy the DAG without writing code.

Performance monitoring is handled by an in‑house tracing system that records per‑service latency for every request, enabling quick bottleneck identification.

Because many inference services are experimental and change frequently, we created PyInter, a custom Python interpreter that runs inside a multi‑threaded process while isolating the interpreter state per request, thus avoiding the Global Interpreter Lock (GIL) and sharing GPU memory across threads.

Benchmarks show that PyInter can achieve comparable or higher QPS than ONNX Runtime while reducing GPU memory usage by up to 80 % when many model replicas are deployed.

Load‑balancing is improved by moving beyond simple random selection. We first use the Power‑of‑2‑Choices algorithm, then enhance it with a Joint‑Idle‑Queue (JIQ) scheme that maintains an idle‑queue and an “amnesia” list of recently active workers, selecting the least‑latency worker from these pools.

In production the JIQ algorithm reduces the P99/P50 latency ratio to 1.5×, a ten‑fold improvement over pure random scheduling.

In summary, the three‑fold solution consists of automated development and deployment pipelines, performance‑aware model serving (including PyInter for experimental services), and a JIQ‑based dynamic scheduler that mitigates long‑tail latency.