Securing Public‑Facing Kafka: Authentication, Configuration, and Scaling Strategies

Why Public Kafka Must Be Secure

Kafka carries over 90% of G7’s internal asynchronous traffic, and exposing it to the public network raises security concerns that involve encryption and AAA (Authentication, Authorization, Auditing).

Encryption and AAA Overview

Transmission Encryption

Plaintext

Symmetric encryption: DES, AES

Asymmetric encryption: SSL/TLS

AAA

Authentication – proving identity (password, fingerprint, facial or iris recognition).

Authorization – defining permissions (e.g., ACL tables).

Audit – recording user actions for traceability.

Kafka Authentication Options

Three schemes are evaluated:

SASL_PLAINTEXT + GSSAPI (Kerberos)

Used in production, but KDC performance and DDoS risk in public environments.

Requires exposing KDC ports, increasing vulnerability.

Requires Kafka ≥ 0.9.

SASL_SSL + PLAIN (PKI)

Passwords stored in plain text configuration – security risk.

No third‑party authentication service exposure.

Requires custom plugin for user creation; Kafka ≥ 0.10.

SASL_SSL + SCRAM (PKI)

Password hashes stored in Zookeeper.

No third‑party service needed.

More complex than option 2; Kafka ≥ 0.10.

Comparing the three, the Kerberos scheme is excluded due to centralised authentication risks, and SCRAM is dropped for its complexity, leaving option 2 (SASL_SSL + PLAIN) as the chosen solution.

Public Proxy Configuration

Key settings include listeners and advertised.listeners to bind broker ports and advertise reachable addresses, e.g.:

listeners=EXTERNAL://0.0.0.0:9093,INTERNAL://0.0.0.0:9098,CONTROLLER://0.0.0.0:9099

advertised.listeners=EXTERNAL://kafka.huoyunren.com:9101,INTERNAL://kafka1.chinawayltd.com:9098,CONTROLLER://kafka1.chinawayltd.com:9099

Additional mappings:

listener.security.protocol.map=EXTERNAL:SASL_SSL,INTERNAL:SASL_PLAINTEXT,CONTROLLER:SASL_PLAINTEXT

inter.broker.listener.name=INTERNAL

control.plane.listener.name=CONTROLLER

sasl.mechanism.inter.broker.protocol=PLAIN

ssl.endpoint.identification.algorithm=

Scaling for Massive Topics

Supporting hundreds of thousands of topics stresses Zookeeper and leads to I/O fragmentation. G7’s solution:

Deploy Kafka on Kubernetes, creating multiple small clusters (≈ 2000 topics each).

Provide an SDK that abstracts multi‑cluster connections for users.

Package a sharded client library to avoid version conflicts.

Extending Data Retention

Default retention is 7 days; customers require much longer periods. G7 uses high‑capacity servers with soft RAID0 across several disks, achieving the following performance:

Metric

Single Disk

Soft RAID0

Random Write IOPS

335

1330

Random Read IOPS

224

757

Sequential Write BW

213 MiB/s

829 MiB/s

Sequential Read BW

215 MiB/s

827 MiB/s

gmq Integration

gmq is G7’s self‑built Kafka management platform handling topic requests, authentication, and monitoring. For the chosen SASL_SSL scheme, gmq was extended to:

Create authentication users.

Deliver AK/SK to user administrators.

Process topic production/consumption authorisation.

Kafka‑REST Considerations

Confluent’s rest‑proxy requires separate security configurations for broker‑to‑rest and client‑to‑rest paths, leading to privilege escalation risks. Mitigations include limiting LB endpoints to only consumer and POST /topics, and using UUID‑based topic names to isolate users.

These practices demonstrate how G7 successfully deployed a secure, public‑facing Kafka service while addressing scalability and operational challenges.