How Pepperbus Unifies Asynchronous Task Management Across Diverse Tech Stacks

Bus systems are common in many businesses, but Huajiao needed a single, unified asynchronous processing platform that could serve multiple teams with different technology stacks. Existing solutions based on Redis, NSQ, or Kafka were fragmented, and the goal was to provide a centralized queue resource management, operation, and monitoring service without building a new queue from scratch.

Requirement Analysis

Make all sub‑systems and micro‑services' asynchronous tasks measurable, controllable, low‑cost, and scalable with transparent scaling.

Enable easy debugging of tasks from creation through consumption and processing, with clear error reporting.

Allow unified integration and management across the company, avoiding separate queue systems and isolated operations.

Provide a PHP‑friendly solution, since the primary online queue service uses PHP with a configuration‑plus‑process‑management framework (processWorker) backed by Redis; Java and Go also need support.

Pepperbus Design

Design Overview

Storage is plug‑in based: any underlying store can be used as long as the application layer offers a unified interface.

The entry communication protocol is Redis protocol, chosen for its maturity across languages, supporting long/short connections, connection pools, and error handling; task production uses the Redis "add job" command.

Task management includes process‑level consumption with timing metrics and built‑in process‑management SDK. For PHP, direct FastCGI communication is used so that task processing time equals the FastCGI response time, allowing seamless code updates via php‑fpm.

For other languages, an HTTP interface forwards tasks, enabling debugging via simple curl commands. A rich dashboard provides task CRUD, troubleshooting, replay, alerts, and monitoring.

The dashboard is business‑driven, offering queue management, troubleshooting, replay, and monitoring to reduce complexity and deliver actionable insights.

System Architecture Diagram

Architecture Description

Business servers can be co‑located with a bus instance or deployed separately; mixed deployment prefers local bus access, while separate deployment uses domain‑based communication.

The bus server interacts with php‑fpm either locally or over the network.

Business Process Flow

Business server pushes a task to the bus service.

The bus stores the task in the chosen storage backend.

The task is handed to a CGI process.

The CGI process returns the processing result.

The bus removes the completed task.

Project Dependencies

Dashboard: provides a web UI for user management.

gokeeper: offers configuration management services.

User Demonstration

Product Requirement Review

Overall backend status can be categorized by system.

Various screenshots illustrate queue list management, real‑time monitoring with replay and gray‑scale replay, error log inspection, testing dashboard that merges historical online data, and storage monitoring for clusters (supporting Pika and Redis) with transparent scaling.

The team completed the core bus development in less than a month.

GitHub contributors: @youlu-cn, @specode, @qudexin1986, @johntech-o