Securing Kafka with Kerberos and ACLs: A Practical Guide

Previous article described ransomware attacks on Hadoop clusters and highlighted the exposure of many Hadoop ports.

Overview Kafka is a high‑throughput distributed publish‑subscribe messaging system originally designed by LinkedIn, written in Scala, widely used for its scalability, reliability and asynchronous communication. It is often paired with Spark Streaming as a mainstream streaming architecture.

Because Kafka lacks built‑in security mechanisms, unauthenticated access can lead to data leakage. This article introduces the Kerberos authentication protocol, explains Kafka’s architecture and key concepts, analyzes existing security issues, and presents Transwarp’s solutions.

Kafka security status Basic concepts:

Topic – a named category of messages.

Producer – process that publishes messages to a topic.

Consumer – process that subscribes to a topic.

Broker – server that stores topics and serves producers and consumers.

Typical Kafka cluster architecture:

Identified security problems:

Any host can start a broker and join the cluster, intercepting or modifying messages.

Any host can launch malicious producers/consumers to send illegal messages or read private data.

Broker does not support Kerberos‑protected ZooKeeper, allowing unrestricted access to ZooKeeper data.

Topic ACLs are absent, so any client can read or write any topic.

These issues become critical in privacy‑sensitive scenarios such as video surveillance.

How to make Kafka more secure?

Transwarp enhances security in two dimensions:

Authentication – implements Kerberos‑based and IP‑address‑based mechanisms.

Authorization – introduces a topic‑level permission model with READ, WRITE, CREATE, DELETE rights.

Kerberos authentication flow (illustrated below) and description of broker, producer, and consumer steps:

Topic ACLs are stored in ZooKeeper under /acl/<topic>/<user> with permission bits (R, W, C, D).

Building a secure Kafka service

Enable Kerberos in broker configuration (server.properties). Example parameters:

Sample Java code for a secure producer using Kerberos:

public class SecureProducer extends Thread {
    private final kafka.javaapi.producer.Producer<Integer, String> producer;
    private final String topic;
    private final Properties props = new Properties();

    public SecureProducer(String topic) {
        AuthenticationManager.setAuthMethod("kerberos");
        AuthenticationManager.login("producer1", "/etc/producer1.keytab");
        props.put("serializer.class", "kafka.serializer.StringEncoder");
        props.put("metadata.broker.list", "172.16.1.190:9092,172.16.1.192:9092,172.16.1.193:9092");
        producer = new kafka.javaapi.producer.Producer<Integer, String>(new ProducerConfig(props));
        this.topic = topic;
    }
}

Topic permission management via AuthorizationManager (illustrated below) with methods resetPermission, grant, revoke, isPermitted, and commit:

Example code for Kerberos‑mode authorization setup:

public class AuthzTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.setProperty("authentication", "kerberos");
        props.setProperty("zookeeper.connect", "172.16.2.116:2181,172.16.2.117:2181,172.16.2.118:2181");
        props.setProperty("principal", "kafka/host1@TDH");
        props.setProperty("keytab", "/usr/lib/kafka/config/kafka.keytab");
        ZKConfig config = new ZKConfig(props);
        AuthenticationManager.setAuthMethod(config.authentication());
        AuthenticationManager.login(config.principal(), config.keytab());
        AuthorizationManager authzManager = new AuthorizationManager(config);
        authzManager.resetPermission("172.16.1.87", new Permissions(Permissions.READ, Permissions.WRITE), "test");
        authzManager.grant("172.16.1.87", new Permissions(Permissions.CREATE), "test");
        authzManager.revoke("172.16.1.87", new Permissions(Permissions.READ), "test");
        authzManager.commit();
        authzManager.close();
    }
}

IP‑address mode uses similar API calls without Kerberos login:

public class AuthzTest {
    public static void main(String[] args) {
        Properties props = new Properties();
        props.setProperty("authentication", "ipaddress");
        props.setProperty("zookeeper.connect", "172.16.1.87:2181,172.16.1.88:2181,172.16.1.89:2181");
        ZKConfig config = new ZKConfig(props);
        AuthorizationManager authzManager = new AuthorizationManager(config);
        authzManager.resetPermission("172.16.1.87", new Permissions(Permissions.READ, Permissions.WRITE), "test");
        authzManager.grant("172.16.1.87", new Permissions(Permissions.CREATE), "test");
        authzManager.revoke("172.16.1.87", new Permissions(Permissions.READ), "test");
        authzManager.commit();
        authzManager.close();
    }
}

Conclusion This article described Kafka’s architecture, exposed its security weaknesses, and detailed Transwarp’s Kerberos authentication and ACL‑based authorization solutions, while noting broader security improvements across the Hadoop & Spark ecosystem.