Master Kafka: Core Concepts and Performance Testing Strategies

Kafka is a high‑performance distributed streaming platform designed for real‑time data processing, stream distribution, and massive message handling, widely used in log collection, real‑time analytics, and event‑driven architectures.

9.1 Kafka Protocol Basics

The architecture enables easy handling of high concurrency and large data volumes. Understanding its core concepts helps test engineers design efficient performance test cases.

Topics : Logical categories for messages; producers and consumers interact via topics, similar to database tables. Example topics: orders for order messages and logs for system logs. Topics support multiple partitions for parallel processing and load balancing.

Partitions : Sub‑units of a topic distributed across servers to improve performance and scalability. Messages in a partition are ordered and each has a unique offset. The number of partitions should match workload and hardware resources.

Producers : Generate and push messages to a topic, optionally specifying a partition. For example, an order service sends order events to the orders topic, including order ID and amount. Producers support asynchronous sending and callbacks for verifying delivery.

Consumers : Pull messages from topics, typically organized in consumer groups that share partition consumption. For instance, an inventory service consumes the orders topic to update stock, while a recommendation service also subscribes to the same topic for personalized suggestions.

Broker and Cluster : A Kafka cluster consists of multiple brokers, each storing partitions of topics. The cluster provides high availability and data redundancy; a 3‑node cluster with replication ensures no data loss.

Offsets : Unique identifiers for messages within a partition. Consumers track offsets to avoid duplicate or missed processing, with support for automatic or manual commits.

ZooKeeper : Earlier Kafka versions relied on ZooKeeper for metadata management and leader election. Since version 2.8, Kafka introduced KRaft mode to remove this dependency, simplifying operations.

Producer workflow steps:

Configure Kafka connection : Set broker address, ports, and producer parameters such as acks, batch.size, and compression.type. Example: acks=all ensures reliable delivery at the cost of higher latency.

Create producer object : Initialize a KafkaProducer instance, e.g., connecting to localhost:9092.

Construct message object : Specify target topic, optional partition, and payload. Keys determine partition selection; values contain business data, e.g., sending order details to the orders topic.

Send message and handle result : Send synchronously or asynchronously, using callbacks to record offset and partition information for performance analysis.

Consumer workflow steps:

Configure Kafka connection : Set broker address, consumer group ID, and deserializer, e.g., group.id=FunTesterGroup.

Create consumer object : Initialize a KafkaConsumer instance.

Subscribe to topics : Subscribe to one or more topics, such as FunTesterTopic, supporting regex patterns.

Poll and process messages : Use the poll method to fetch messages periodically and execute business logic, like updating a database with order information.

Commit offsets : Commit offsets manually or automatically to record consumption progress, useful for strict consistency requirements.

In performance testing, producers can simulate massive user‑generated data (orders, logs, events) to evaluate Kafka’s push performance, while consumers verify processing speed and correctness under high load, checking for missed or duplicate messages. Test engineers should monitor broker load, partition distribution, and network latency to ensure stability in production scenarios.

To enable Kafka performance testing, add the following Maven dependency to provide the Kafka client library:

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-clients</artifactId>
  <version>3.4.0</version>
</dependency>