Setting Up the ELK Stack with Docker Compose for Log Collection and Analysis

Elasticsearch is an open‑source distributed search engine that offers data collection, analysis, and storage capabilities.

Logstash is primarily used for log collection, analysis, and filtering, supporting a wide range of data ingestion methods.

Kibana is also an open‑source, free tool that provides a user‑friendly web interface for Logstash and Elasticsearch logs, helping to aggregate, analyze, and search important data logs.

In simple terms, application services generate logs via a logger; Logstash receives the logs over HTTP; Elasticsearch provides full‑text search for the logs; Kibana offers a graphical interface for Elasticsearch.

version: '3.4'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
    restart: always
  kafka:
    image: wurstmeister/kafka:2.3.0
    volumes:
      - /etc/localtime:/etc/localtime
    ports:
      - "9092:9092"
    environment:
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_HOST_NAME=192.168.20.25
      - KAFKA_ADVERTISED_PORT=9092
      - KAFKA_LOG_RETENTION_HOURS=120
      - KAFKA_MESSAGE_MAX_BYTES=10000000
      - KAFKA_REPLICA_FETCH_MAX_BYTES=10000000
      - KAFKA_GROUP_MAX_SESSION_TIMEOUT_MS=60000
      - KAFKA_NUM_PARTITIONS=3
      - KAFKA_DELETE_RETENTION_MS=1000
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    restart: always
  kafka-manager:
    image: sheepkiller/kafka-manager
    environment:
      ZK_HOSTS: 192.168.20.25
    ports:
      - "9001:9000"
  elasticsearch:
    image: daocloud.io/library/elasticsearch:7.6.2
    restart: always
    container_name: elasticsearch
    environment:
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ports:
      - 9200:9200
  kibana:
    image: daocloud.io/library/kibana:7.6.2
    restart: always
    container_name: kibana
    ports:
      - 5601:5601
    environment:
      - elasticsearch_url=http://192.168.20.25:9200
    depends_on:
      - elasticsearch

To start the services, run: docker-compose up -d If startup errors occur, first stop and remove the containers before restarting: docker-compose down If this article helped you, please like, view, and share—it’s crucial for me to continue sharing and creating quality content. Thank you 🙏🏻

More practical video courses are also available for further learning.