Comparing Kafka and Mosquitto for Microservice Communication

Communication between Microservices

In recent years, microservice architecture has become increasingly popular, and orchestration tools like Docker and Kubernetes make creating and maintaining such applications very easy, even driving the adoption of microservices.

Unfortunately, this architecture brings challenges, one of which is finding a good way for microservices to communicate.

If you face this issue, you have likely heard of three solutions: REST API, MQTT, or Kafka. In this use case, REST API is not feasible, leaving MQTT and Kafka as the remaining options.

Both are messaging brokers with different protocols and purposes, so let's compare them.

Perfect Messaging Broker

Let's start with what matters most for microservices.

Microservices need persistent data storage, must handle large traffic and large data sets, be able to split traffic into independent logical parts such as topics, and require high reliability and event delivery capabilities.

Considering these requirements, we compare Kafka and Mosquitto.

Kafka

Kafka was originally created by LinkedIn engineers in 2011 for their information system and later released under the Apache license. Kafka persists events, meaning messages are written to the file system as soon as they are received.

It scales services without worrying about duplicate event processing. Kafka expands by adding more partitions, allowing messages from each partition to be processed in parallel, making it highly scalable because it was built for large systems from the start.

Kafka can easily handle high‑speed data ingestion. Moreover, its libraries are written in popular languages, and it offers interesting frameworks such as the Ruby‑based Karafka.

Unfortunately, Kafka relies on Zookeeper to operate correctly.

Mosquitto

Now let's look at Mosquitto, another tool for microservice communication.

Mosquitto is one of the most popular solutions, a lightweight protocol created for IoT projects. It follows a publish/subscribe model, and the message broker operates independently of other applications or libraries.

Mosquitto is licensed under EPL/END, making it open source and part of the Eclipse Foundation—an important factor for many projects. It offers libraries in multiple languages, making it highly versatile and easy for developers to adapt to their projects.

Answer

Kafka is more suitable for microservices. Its persistent storage enables replay of events from specific topics, supporting an event‑sourcing pattern.

Source: http://jiagoushi.pro/node/1101

Discussion: Join the Knowledge Planet "Chief Architect Circle" or the small account "jiagoushi_pro".