Kafka Main Configuration Parameters – Broker, Producer, Consumer, and Topic Settings

Kafka Main Configuration

6.1 Broker Config

Property

Default

Description

broker.id

Required unique identifier for the broker.

log.dirs

/tmp/kafka-logs

Directories where Kafka stores data; multiple directories can be specified, separated by commas. New partitions are placed in the directory with the fewest existing partitions.

port

9092

Port on which the broker accepts client connections.

zookeeper.connect

null

Zookeeper connection string (host:port pairs). It can include a chroot path to isolate the Kafka cluster data.

message.max.bytes

1000000

Maximum size of a message the broker will accept; should match the consumer's maximum.message.size.

num.io.threads

8

Number of I/O threads for handling read/write requests; should be at least the number of disks.

queued.max.requests

500

Maximum number of requests that I/O threads can queue; excess requests cause the network thread to stop accepting new ones.

socket.send.buffer.bytes

100 * 1024

Preferred socket send buffer size (SO_SNDBUF).

socket.receive.buffer.bytes

100 * 1024

Preferred socket receive buffer size (SO_RCVBUF).

socket.request.max.bytes

100 * 1024 * 1024

Maximum size of a request the broker will accept to prevent memory overflow; should be less than the Java heap size.

num.partitions

1

Default number of partitions per topic if not specified; recommended to increase to 5.

log.segment.bytes

1024 * 1024 * 1024

Size of a log segment file; a new segment is created when this size is exceeded. Topic‑level settings can override it.

log.roll.{ms,hours}

24 * 7 hours

Time interval after which a new segment is rolled; can be overridden per topic.

log.retention.{ms,minutes,hours}

7 days

Retention period for log segments; segments older than this are deleted. Topic‑level overrides are possible.

log.retention.bytes

-1

Maximum size per partition; -1 means unlimited. Log‑level settings can override it.

log.retention.check.interval.ms

5 minutes

Frequency at which the broker checks for log retention violations.

auto.create.topics.enable

true

Whether topics are created automatically; recommended to set to false for stricter management.

default.replication.factor

1

Default number of replicas for a topic; recommended to set to 2.

replica.lag.time.max.ms

10000

Time window after which a follower that hasn't fetched data is removed from the ISR.

replica.lag.max.messages

4000

Maximum number of messages a lagging replica can be behind before removal from ISR.

replica.socket.timeout.ms

30 * 1000

Timeout for replica requests to the leader.

replica.socket.receive.buffer.bytes

64 * 1024

Socket receive buffer size for replica‑to‑leader communication.

replica.fetch.max.bytes

1024 * 1024

Maximum bytes a replica will fetch per partition request.

replica.fetch.wait.max.ms

500

Maximum wait time for data to become available on the leader for replica fetches.

num.replica.fetchers

1

Number of threads used by a broker to replicate data from leaders.

fetch.purgatory.purge.interval.requests

1000

Number of fetch requests after which the fetch‑purgatory is purged.

zookeeper.session.timeout.ms

6000

Zookeeper session timeout; if the broker fails to heartbeat within this period, it is considered dead.

zookeeper.connection.timeout.ms

6000

Timeout for establishing a connection to Zookeeper.

zookeeper.sync.time.ms

2000

Maximum lag time for a ZK follower behind the leader.

controlled.shutdown.enable

true

Enables controlled shutdown, allowing a broker to transfer leadership before exiting.

auto.leader.rebalance.enable

true

Allows the controller to automatically rebalance partition leadership across brokers.

leader.imbalance.per.broker.percentage

10

Maximum allowed leader imbalance percentage per broker before triggering a rebalance.

leader.imbalance.check.interval.seconds

300

Interval at which leader imbalance is checked.

offset.metadata.max.bytes

4096

Maximum metadata size that clients can store with their offsets.

connections.max.idle.ms

600000

Idle connection timeout; idle sockets are closed after this period.

num.recovery.threads.per.data.dir

1

Number of threads per data directory for log recovery at startup and flushing at shutdown.

unclean.leader.election.enable

true

Allows non‑ISR replicas to be elected as leader as a last resort, risking data loss.

delete.topic.enable

false

Whether topics can be deleted; recommended to set to true.

offsets.topic.num.partitions

50

Number of partitions for the internal offsets topic; higher values improve production scalability.

offsets.topic.retention.minutes

1440

Retention period for offset data; older offsets are marked for deletion.

offsets.retention.check.interval.ms

600000

Frequency at which the offset manager checks for stale offsets.

offsets.topic.replication.factor

3

Replication factor for the offsets topic; higher values increase availability.

offsets.topic.segment.bytes

104857600

Segment size for the offsets topic; kept relatively low for faster compaction.

offsets.load.buffer.size

5242880

Batch size (in bytes) used when loading offsets into the manager's cache.

offsets.commit.required.acks

-1

Number of acknowledgments required before an offset commit is accepted.

offsets.commit.timeout.ms

5000

Timeout for offset commit; the commit is delayed until this timeout or required replicas acknowledge.

6.2 Producer Config

Property

Default

Description

metadata.broker.list

List of broker hosts (host:port) that the producer contacts to obtain metadata.

request.required.acks

0

Acknowledgment requirement for producer requests (see section 3.2).

request.timeout.ms

10000

Timeout for the broker to acknowledge a request; exceeds this results in an error.

producer.type

sync

Producer mode: "sync" for synchronous, "async" for asynchronous (batching). Async is recommended for higher throughput.

serializer.class

kafka.serializer.DefaultEncoder

Class used to serialize messages; defaults to byte[].

key.serializer.class

Serializer for message keys; defaults to the same as serializer.class.

partitioner.class

kafka.producer.DefaultPartitioner

Class that determines partition assignment; default hashes the key.

compression.codec

none

Compression algorithm for messages: "none", "gzip", or "snappy".

compressed.topics

null

Comma‑separated list of topics to compress; if null, all topics are compressed when a codec is set.

message.send.max.retries

3

Number of retry attempts for failed sends.

retry.backoff.ms

100

Delay before retrying after a failure, allowing metadata refresh.

topic.metadata.refresh.interval.ms

600000

Interval for periodic metadata refresh; negative disables periodic refresh.

queue.buffering.max.ms

5000

Maximum time to buffer messages in async mode before sending.

queue.buffering.max.messages

10000

Maximum number of messages that can be buffered in async mode.

queue.enqueue.timeout.ms

-1

Time the producer blocks when the buffer is full; -1 means block indefinitely.

batch.num.messages

200

Number of messages per batch in async mode before sending.

send.buffer.bytes

100 * 1024

Socket write buffer size for the producer.

client.id

""

Identifier string sent with each request for tracing.

6.3 Consumer Config

Property

Default

Description

group.id

Consumer group identifier; consumers with the same group.id belong to the same group.

zookeeper.connect

Zookeeper connection string for the consumer; must match the broker configuration.

consumer.id

null

Explicit consumer identifier; if not set, a random ID is generated.

socket.timeout.ms

30 * 1000

Socket timeout for network requests; actual timeout also depends on max.fetch.wait.

socket.receive.buffer.bytes

64 * 1024

Socket receive buffer size for network requests.

fetch.message.max.bytes

1024 * 1024

Maximum size of a message the consumer will fetch per partition; should be larger than the broker's max message size.

num.consumer.fetchers

1

Number of fetcher threads used by the consumer.

auto.commit.enable

true

If true, the consumer periodically commits its offset to Zookeeper.

auto.commit.interval.ms

60 * 1000

Interval between automatic offset commits.

queued.max.message.chunks

2

Number of message chunks the consumer can buffer; each chunk can hold fetch.message.max.bytes of data.

rebalance.max.retries

4

Maximum number of rebalance attempts before giving up.

fetch.min.bytes

1

Minimum amount of data the server should return for a fetch request.

fetch.wait.max.ms

100

Maximum time the server will block waiting for enough data to satisfy fetch.min.bytes.

rebalance.backoff.ms

2000

Backoff time between rebalance retries.

refresh.leader.backoff.ms

200

Backoff time before retrying to find a new leader for a partition that lost its leader.

auto.offset.reset

largest

Behavior when no initial offset is found: "smallest" resets to earliest offset, "largest" to latest offset.

consumer.timeout.ms

-1

Timeout after which the consumer throws an exception if no messages are received.

exclude.internal.topics

true

Whether to hide internal topics (e.g., offsets) from the consumer.

zookeeper.session.timeout.ms

6000

Zookeeper session timeout for the consumer.

zookeeper.connection.timeout.ms

6000

Maximum time to wait while establishing a connection to Zookeeper.

zookeeper.sync.time.ms

2000

Maximum lag a Zookeeper follower can have behind the leader.

6.4 Topic‑Level Configuration

Topic‑specific configuration options are documented at Kafka Topic Config . These settings allow fine‑grained control over retention, cleanup, compression, and other behaviors on a per‑topic basis.