Mastering Log Collection: From Daily Ops to the ELK Stack

What Do Ops Spend Their Time On?

Typical daily scenarios show developers and testers asking ops to check logs, retrieve log files, or investigate errors, while ops spend a large portion of their time handling log‑related tasks such as deployment, script execution, and log analysis.

Log Requirements

System logs : Monitor OS health, detect hardware failures, and capture low‑level events.

Access logs : Gather traffic sources, request frequencies, response times, and success rates for business analysis.

Runtime logs : Record application behavior, exceptions, and business‑level output.

Error logs : Search by keywords to correlate errors with traffic spikes and perform full‑traffic analysis.

Business‑related logs : Correlate orders, promotions, and other domain events to answer questions like “why did orders surge in the last five minutes?”.

Challenges in Log‑Collection Environments

Developers need log access but often lack permission to log into production servers, and many large companies restrict such access. Logs are scattered across heterogeneous systems, making collection and standardization difficult.

Solving the Pain Points: The ELK Architecture

Legacy ELK Architecture

The classic ELK stack consists of Logstash, Elasticsearch, and Kibana. In containerized environments, a Docker instance runs on each host, Logstash gathers logs from container directories, and the stack operates with low cost and minimal technical barriers.

Current Elastic Stack

Renamed to Elastic Stack after Elastic acquired Beats, the ecosystem now includes Beats for log, network, and stream collection, expanding its capabilities. Elastic also offers SaaS solutions for broader internet users.

ELK Learning: Elasticsearch Introduction

Elasticsearch clusters have three health states: green (all primary and replica shards active), yellow (all primaries active but some replicas missing), and red (primary shards failing, risking data loss). An index defaults to five primary shards and one replica; replicas provide redundancy but not performance gains. When a node fails, Elasticsearch automatically elects a new master and rebalances shards.

ELK Learning: Logstash Introduction

Logstash processes data through three stages: input, filter, and output. A simple "hello world" reads from stdin, adds a timestamp, and outputs to stdout. Common use cases include reading system log files via input plugins and sending them to Elasticsearch via output plugins. Logstash also supports message‑queue outputs (e.g., Kafka, RabbitMQ) for decoupled, reliable log pipelines.

ELK Learning: Kibana Overview and Enterprise Practice

Kibana Introduction

Kibana visualizes data stored in Elasticsearch. Users can select time ranges, save searches, create visualizations, and assemble dashboards for operational monitoring, such as user registration rates, active users, and database logs.

Enterprise Practices

Effective log management focuses on standardizing log formats before ingestion. Standardized logs enable reliable collection, alerting based on keyword matches, and meaningful analysis across services. Logstash can also trigger alerts when specific patterns appear, helping teams quickly identify issues like sudden traffic drops or error spikes.