How to Build a Scalable Task System with Kafka and Pub/Sub

Functional Division

The task system consists of four core functions: task configuration (dynamic settings for text, rewards, activation), task display (status, ordering, categorization), task progress (updating progress when users perform actions such as forming a team or sending gifts), and reward claiming (automatic or manual distribution after completion).

Call Relationship

When a user completes an action (e.g., the first team formation), the game service must notify the task service to update progress. Direct API calls create tight coupling, especially when multiple services need the same notification.

Using Kafka for Decoupling

Kafka, a high‑throughput distributed message queue, separates producers and consumers. After the main logic finishes, services publish messages to Kafka; interested services subscribe to relevant topics and receive notifications without modifying the producer.

Event Handling Layer

To avoid embedding task logic in consumer code, an event handling layer abstracts event sources from task processing. It shields the task layer from source details and allows tasks to subscribe to events they care about.

Publish/Subscribe Pattern

The pattern (also known as the observer pattern) further decouples publishers from subscribers: publishers emit events without knowing who receives them, and subscribers can be added or removed without changing publisher code.

Task Configuration

Task attributes such as description, reward, and navigation actions are stored in a database, allowing dynamic updates without code changes. The task logic remains in code and references tasks by ID.

Task Classification

Beyond UI categories, tasks are classified at the entity level as ordinary, periodic, or continuous tasks, supporting reuse of logic for similar task types such as newbie tasks, daily tasks, and sign‑in streaks.

Summary

Use Kafka to decouple inter‑service communication.

Introduce an event handling layer to separate task logic from event sources.

Separate task attributes from logic for dynamic configuration.

Apply layered classification to enable task type reuse.