Mastering TDMQ CKafka Production: Configurations, Partitioning, and Reliable Messaging

Overview

TDMQ CKafka is a distributed, high‑throughput, Kafka‑compatible message system. For developers, mastering its producer configuration is essential to build stable, high‑performance data pipelines.

Topic Creation and Configuration

Topic usage and creation : Topics should be created with a replica count that is a multiple of the node count to avoid data skew. The minimum in‑sync replica count is 2, and the replica count must not equal the topic replica count, otherwise a single node failure blocks production.

Creation mode : CKafka can automatically create a topic when a producer references a non‑existent one. The default auto‑created topic contains three partitions and two replicas.

Partition Sizing

Estimate partitions as a multiple of the number of nodes. Use a rule of thumb of 10 MB/s per partition; e.g., a 100 MB/s throughput suggests 10 partitions.

Retry Strategy

Retries : Number of retry attempts (default 3). Increase if message loss is unacceptable.

Retry.backoff.ms : Interval between retries, recommended 1000 ms.

These settings help handle broker leader unavailability.

Asynchronous Sending

Producers send messages asynchronously. To obtain send results, use the Send method’s callback interface.

Producer Instance per Application

Producer clients are thread‑safe; a single producer per application is recommended.

ACK Settings

The acks parameter controls how many broker acknowledgments the producer waits for: acks=0: No acknowledgment, highest throughput, risk of data loss. acks=1: Leader acknowledgment only, balanced performance and reliability. acks=all: All in‑sync replicas must acknowledge, highest safety but lower throughput.

For most cases, use acks=1; critical services may require acks=all.

Batch Tuning

Messages sent to the same partition are grouped into batches. Small batches increase request overhead and latency. Key parameters:

batch.size : Buffer size per partition before a request is triggered.

linger.ms : Maximum time to wait before sending a batch regardless of size.

buffer.memory : Total buffer memory (default 32 MB).

Key and Value Handling

Setting a unique key enables tracking and ordering. For high‑volume streams, avoid keys or use a well‑distributed key strategy to prevent partition skew.

Sticky Partitioning

When no key is provided, CKafka (Kafka ≥ 2.4) uses a sticky partitioning strategy: after a batch completes, a new partition is randomly selected and subsequent messages are sent there until the next batch completes. This reduces the number of small batches while keeping long‑term load balanced.

Custom partitioners can be implemented via Partitioner.class. Example Java sticky partitioner is provided.

Ordering Guarantees

Within a single partition, messages retain order. To achieve global order, set the topic to a single partition; for partition‑level order, ensure messages share the same key.

Parameter Best Practices for Ordered Scenarios

enable.idempotence : Set to true (required for Kafka ≥ 0.11).

acks : Must be all when idempotence is enabled.

max.in.flight.requests.per.connection : Set to 1 for Kafka 0.11‑1.0; 1‑5 (recommended 5) for Kafka ≥ 1.1.

retries : Set to Integer.MAX_VALUE for ordered scenarios.

Example property settings:

// create Producer properties
Properties properties = new Properties();
properties.setProperty("enable.idempotence", "true");
properties.setProperty("acks", "all");
properties.setProperty("max.in.flight.requests.per.connection", "5");
properties.setProperty("retries", Integer.toString(Integer.MAX_VALUE));

Data Skew Mitigation

Skew arises from uneven partition distribution or key distribution. Mitigation strategies include:

Choose a partition count that is a multiple of the node count.

Use balanced partitioning strategies (RoundRobin, Range, Sticky, or custom).

Design keys to distribute evenly across partitions.

Conclusion

Effective CKafka production requires careful topic design, retry tuning, idempotence, ACK configuration, and batch optimization. Applying these settings yields a reliable, low‑latency message pipeline and avoids common pitfalls such as data skew and out‑of‑order delivery.