Build a Log Analysis Platform in 30 Minutes with ELK, Ansible, and Vagrant

Overview

Log analysis platforms transform raw web server logs (e.g., Nginx) into structured data for querying user behavior, security events, and business metrics. The ELK stack—Elasticsearch, Logstash, and Kibana—enables small teams to process gigabytes of logs per day, especially when combined with automation tools such as Ansible.

Goal and Source Repository

The objective is to provide a minimal, extensible log‑analysis platform. The complete source code is available at http://git.oschina.net/zacker330/OSC-AdCenter.

Architecture

OpenResty : Nginx‑based web platform that can be extended with Lua scripts.

Logstash : Collects logs, parses them into structured JSON, and forwards the data to Elasticsearch. Alternative collectors (e.g., Beats) can be used.

Elasticsearch : Distributed search engine built on Lucene, stores the indexed log documents.

Kibana : Node.js‑based UI for visualizing and querying the indexed data.

Required Tools

Ansible 2.0+ – automation and configuration management.

Vagrant – local virtual machine manager (VirtualBox provider).

A text editor with YAML syntax highlighting (e.g., Atom).

JDK 8 – download jdk-8u66-linux-x64.tar.gz and place it in provision/roles/jdk8/files/.

Provision a Development VM

Clone the repository, then run vagrant up. The Vagrantfile defines the VM:

Vagrant.configure(2) do |config|
  ANSIBLE_RAW_SSH_ARGS = []
  machine_box = "trusty-server-cloudimg-amd64-vagrant-disk1"
  machine_box_url = "https://cloud-images.ubuntu.com/vagrant/trusty/current/trusty-server-cloudimg-amd64-vagrant-disk1.box"
  config.vm.define "oscadcenter" do |machine|
    machine.vm.box = machine_box
    machine.vm.box_url = machine_box_url
    machine.vm.hostname = "oscadcenter"
    machine.vm.network "private_network", ip: "192.168.4.10"
    machine.vm.provider "virtualbox" do |node|
      node.name = "oscadcenter"
      node.memory = 4048
      node.cpus = 2
    end
  end
end

Default Vagrant credentials are vagrant/vagrant. SSH into the VM with vagrant ssh oscadcenter.

Deploy the Platform with Ansible

On the host machine, install Ansible and execute a single playbook command:

ansible-playbook ./provision/playbook.yml -i ./provision/inventory -u vagrant -k

You will be prompted for the Vagrant SSH password ( vagrant).

The inventory file maps the host name to the VM IP:

[adcenter]
192.168.4.10

The playbook pulls ELK roles from Ansible Galaxy and performs the following steps:

Common base setup.

Install OpenResty.

Configure OpenResty with a custom analysis.conf.

Install JDK 8 and set JAVA_HOME.

Install Elasticsearch, Kibana, and Logstash.

Relevant role list (excerpt from ./provision/playbook.yml):

- hosts: analysis
  sudo: yes
  vars_files:
    - ./vars/base-env.yml
    - ./vars/analysis-logstash.yml
  roles:
    - common
    - openresty
    - {role: "analysis-openresty-conf", nginx_server_conf: "analysis.conf"}
    - jdk8
    - ansible-role-elasticsearch
    - ansible-role-kibana-4
    - ansible-logstash

Verification

Open Elasticsearch Head UI at http://192.168.4.10:9200/_plugin/head/ to view cluster health.

Open Kibana at http://192.168.4.10:5601 to explore visualizations.

Send a test request, e.g.,

http://192.168.4.10/1.gif?account=oschina&e=pv&p=p233444&url=www.oschina.net&title=learning&sh=1200&sw=800&cd=400&lang=en

, and confirm the log entry appears in Elasticsearch and Kibana.

Next Steps

Learn Kibana query syntax to build business‑specific dashboards.

Implement monitoring for the log‑analysis platform to detect outages.

Integrate the platform with existing systems.

Plan scaling strategies for larger Elasticsearch clusters.

Architecture Diagram