How 58’s Messaging Platform Solved High‑Load Real‑Time Delivery Challenges

My talk today includes three parts: background of the messaging platform, its overall architecture and system focus, and the problems we encountered and solved.

Background of the Messaging Platform

The platform was created for 58’s core business “58 Speed Freight”, which provides convenient intra‑city cargo delivery. The original system used frequent GPS reports and HTTP push, causing high pressure, low availability, and delivery issues, so a dedicated messaging platform was built to improve stability and reduce load on business services.

Overall Architecture

The system serves multiple scenarios: an APP side (users or drivers) that reports GPS and places orders, and a driver side that receives order pushes and can grab orders. The platform also supports other 58 services such as home services and beauty.

Based on this, we designed the architecture shown in Figure 1.

On the APP side, two kinds of messages exist: from the APP to the server and from the server back to the APP.

msg-gate aggregates millions of TCP connections, maintains secure channels, handles encryption, compression, traffic monitoring, black‑/white‑listing, and delivers messages between APP‑server and APP clients. It also converts gateway messages to MQ for downstream processing.

The logic layer is stateless and focuses on business‑related processing, re‑assembling gateway messages for the APP.

System Focus

Although the architecture looks simple, implementation faced two major challenges: protocol design and fast reconnection.

Protocol Design

We designed the protocol with extensibility, debuggability, and asynchronous handling in mind.

Extensibility: support various message types such as login, business, push, keepalive, and allow flexible payloads. Figure 2 illustrates the message format with a command identifier to distinguish request, response, or push messages.

Debuggability: a unified identifier helps trace a message through APP → gate → MQ → logic → Redis, as shown in Figure 3.

Asynchronous support: each message carries an ID (Figure 4) and is processed via a context queue and callbacks to handle out‑of‑order responses.

Fast Reconnection

TCP fast reconnection required keeping the connection alive to avoid unnecessary re‑creation. In unstable wireless networks, if a connection drops, we must decide whether to discard pending pushes or attempt reconnection. We clear the connection for failed pushes and, when a driver logs into multiple gates, we retain the last successful server and reuse it to prevent duplicate logins.

That concludes the introduction to the messaging platform.