Why MaFengWo Built Its Own Message Bus and How It Works

Introduction

MaFengWo's Message Bus launched in November 2017 and is now used in production by teams across e‑commerce, hospitality, transportation, and community services. Over the past year it has undergone several major feature iterations, handling most of the asynchronous, throttling, and decoupling tasks for PHP online services. This article shares the design rationale, implementation principles, and future roadmap.

Why Do We Need a Message Bus?

Before the bus, most asynchronous needs were met with Redis queues, which caused message backlogs and cross‑message interference as volume grew. Directly connecting business systems to RabbitMQ or Kafka would not solve these issues and would introduce new problems:

System fragmentation : each team would maintain its own message service, leading to isolation.

Higher development difficulty : developers must configure and integrate with various message services.

High maintenance cost : each team must ensure consumer stability and handle failures, especially challenging for PHP.

Management complexity : no unified way to manage message creation, subscription, or permission control.

Limited scalability : unified features such as routing, delay, retry, and acknowledgment cannot be applied consistently.

In contrast, a message bus provides a business‑oriented access point that abstracts deployment, grouping, and communication details.

Architecture Design and Technical Implementation

1. Architecture Design

The bus hides sending, routing, grouping, storage, consumption load, communication, and high‑availability concerns. Users simply call an SDK method to publish messages and implement a PHP consumer method to receive them.

The current stack uses RabbitMQ as the message engine. The SDK acts as a broker, handling routing and grouping, while subscription relationships are persisted in MySQL. On the consumer side, a Go‑based Deliver service converts AMQP to HTTP, which the PHP service then executes.

2. Technical Implementation

1) Reduce Flow Complexity

Message bodies follow a unified structure with three parts: Parameter (ID, source, timestamp, etc.), Content (business payload), and Receiver (the PHP consumer method).

2) Online Service Asynchrony (Point‑to‑Point)

Business logic that does not need synchronous execution is placed in an asynchronous flow. The sender specifies a unique Receiver, and the bus’s invoke method routes the message accordingly.

3) Decoupling via Publish‑Subscribe

The publish‑subscribe model provides a standard decoupling mechanism. Subscriptions are stored in a database; when a message passes the broker, it is fanned out to all subscribed receivers.

4) Prevent Message Interference

Messages are routed to an Online Queue. Multiple Deliver workers consume from it, with a Dispatcher limiting overall and per‑type concurrency. Excessive load for a type is redirected to an Offline Queue, processed by separate workers, ensuring one type does not block others.

5) Consumer Service High Availability

Deliver clusters implement several resilience strategies:

Circuit breaking – stops consuming a queue when failure rates exceed a threshold.

Failure recovery – monitors downstream service health and resumes consumption automatically.

System‑level retry – retries messages after bus‑side failures.

Business‑level retry – allows configurable retry policies per subscription.

Graceful restart – ensures in‑flight messages are processed before shutdown.

Future Planning

1. Productization

Although stable after a year of iteration, the bus still lacks user‑friendly management, monitoring, and analytics. Planned improvements include a web UI for creating and subscribing to messages (with permission review), real‑time consumption alerts, and finer‑grained monitoring metrics.

2. Micro‑service Integration

Future work will expose the bus as a micro‑service, supporting multi‑language clients and diverse architectural patterns. Additional message engines such as Kafka and RocketMQ will be selectable based on reliability or throughput requirements, while keeping the same SDK interface for producers and consumers.