Understanding Message Middleware: Core Architecture and Kafka Basics

Message Middleware Basic Architecture

Message middleware can be divided into three parts:

Producer – the source that generates messages and sends them to the queue.

Queue – receives and stores messages from producers.

Consumer – retrieves and processes messages from the queue.

Roles of Message Middleware

The main functions of a message queue are peak shaving, asynchronous processing, and decoupling .

Two Consumption Modes

Messages can be consumed via two patterns: publish‑subscribe (one‑to‑many) and point‑to‑point (one‑to‑one).

Publish‑Subscribe Mode (One‑to‑Many)

After a consumer reads a message, the message is retained for a period and can be consumed by all subscribers. This mode supports two delivery methods:

Push mode : the queue pushes messages to consumers. Faster consumers may handle the load, while slower ones may fall behind.

Pull mode : consumers actively pull messages from the queue at a rate they can handle, requiring each consumer to maintain a pulling task.

Point‑to‑Point Mode (One‑to‑One)

Consumers actively pull data; once a message is acknowledged, the queue deletes it, ensuring each message is processed by only one consumer.

Kafka Basic Concepts

Kafka is a distributed message queue that uses Zookeeper for cluster management.

Different consumer groups operate on a publish/subscribe pull model; within the same consumer group, consumers use a point‑to‑point model, but messages are not deleted after consumption.

Key Kafka Components

Producer : client that sends messages to a Kafka broker.

Broker : a Kafka server; a cluster consists of multiple brokers managed by Zookeeper.

Consumer : client that retrieves messages from a broker.

Consumer Group : a set of consumers; only one consumer in the same group can read a given partition.

Topic : a logical queue that groups messages of the same type, abstracting partitions and replicas.

Partition : the physical storage unit of a topic; a partition can be consumed by multiple consumer groups simultaneously, but only by one consumer within the same group. Increasing partition count boosts a topic’s throughput.

Replica : a backup of a partition. Each partition has one leader and one or more followers (replicas) on different machines. The leader handles producer and consumer interactions; followers replicate data and take over if the leader fails, ensuring availability.

To improve Kafka consumption performance, increase both the number of partitions and the number of consumers in the same consumer group.

An analogy with database sharding: brokers correspond to database instances, partitions to table shards, topics to logical tables, and replicas to read‑only replicas.

Code Example – Sending Messages

@Resource
private KafkaTemplate<String, String> kafkaTemplate;

public <T> void plainNotify(String topic, T obj) {
    try {
        this.kafkaTemplate.send(topic, JSON.toJSONString(obj));
    } catch (Exception e) {
        // TODO
    }
}

Code Example – Consuming Messages

@Service
@Slf4j
public class Consumer {

    @KafkaListener(id = "ConsumerGroupId", topics = {"TOPIC1", "TOPIC2"})
    public void consumer(List<String> msgs, Acknowledgment ack) {
        for (String msg : msgs) {
            // TODO
        }
        ack.acknowledge();
    }
}