Introduction to the ELK Stack and Kafka with Docker Compose

Elasticsearch is a Lucene‑based search server that offers data collection, analysis, and storage, providing a distributed, multi‑user full‑text search engine accessed via a RESTful API. Developed in Java and released under the Apache license, it is cloud‑ready, real‑time, stable, reliable, fast, and easy to install.

Logstash is a tool for searching, analyzing, and filtering logs; it collects, transforms, and parses log data and makes it available to other modules such as search and storage systems.

Kibana is a front‑end visualization framework that converts log data into various charts, tables, and maps, offering powerful data visualization by displaying indexed data stored in Elasticsearch.

Kafka is a data buffering queue that acts as a message broker, decoupling processing steps, enhancing scalability, handling peak loads, and preventing system crashes caused by sudden traffic spikes.

The following Docker‑Compose file defines services for Zookeeper, Kafka, Kafka‑Manager, Elasticsearch, and Kibana, specifying images, ports, environment variables, volumes, restart policies, and service dependencies to enable a ready‑to‑run ELK‑Kafka stack:

version: '3.4'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
    restart: always
  kafka:
    image: wurstmeister/kafka:2.3.0
    volumes:
      - /etc/localtime:/etc/localtime
    ports:
      - "9092:9092"
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_HOST_NAME=192.168.20.235
      - KAFKA_ADVERTISED_PORT=9092
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always
  kafka-manager:
    image: sheepkiller/kafka-manager
    environment:
      ZK_HOSTS: 192.168.20.235
    ports:
      - "9001:9000"
  elasticsearch:
    image: daocloud.io/library/elasticsearch:7.6.2
    restart: always
    container_name: elasticsearch
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - 9200:9200
  kibana:
    image: daocloud.io/library/kibana:7.6.2
    restart: always
    container_name: kibana
    ports:
      - 5601:5601
    environment:
      - elasticsearch_url=http://192.168.20.235:9200
    depends_on:
      - elasticsearch