An Out‑of‑the‑Box ELK‑Based Log and Metric Collection Solution for Private Deployments

In private‑deployment scenarios where systems mainly provide business functions such as user, finance, or customer management, there is still a need to collect logs and application metrics (e.g., request rate, host disk, memory usage). To ensure stability without adding operational overhead, this article proposes an out‑of‑the‑box ELK‑based log and metric collection solution.

1 Background

The project already uses Elasticsearch for business data storage and combines Ansible, Docker, and Jenkins for rapid deployment. After configuring SSH connections, a one‑click Jenkins job can deploy Elasticsearch and Kibana.

2 Requirement Analysis

Fast deployment with minimal operational pressure.

Simple, robust components that avoid complex network topologies.

Feature richness prioritized over high availability; occasional data loss is acceptable.

Low performance requirements allow single‑node deployment.

Key functional requirements include distributed log collection, host resource monitoring (CPU, disk, memory), optional application‑level metrics, and anomaly alerting.

3 Solution Analysis

Three candidate solutions were evaluated:

ELK stack (Elasticsearch, Logstash, Kibana) with Beats for data collection.

Zabbix / Open‑Falcon for system monitoring and custom metrics.

TICK stack (Telegraf, InfluxDB, Chronograf, Kapacitor).

Considering cost and functional fit, the ELK solution was selected and further refined to meet the specific needs.

4 ELK Component Deployment

A dedicated Elasticsearch instance (≈3 GB RAM) is deployed solely for logging to avoid conflicts with business data stores. Filebeat is used on all hosts for log collection, eliminating Logstash for simplicity. Kibana provides direct log visualization.

Metricbeat gathers host metrics (CPU, memory, disk, network) and Docker resource usage, and also exposes a StatsD endpoint for custom application metrics.

Heartbeat runs on gateway machines to probe service availability (HTTP, databases) and stores results in Elasticsearch.

5 Alerting with ElastAlert

Since native Elasticsearch alerting is commercial, the open‑source ElastAlert (Python‑based) is employed for email alerts. Supported alert types include frequency, spike, and flatline, each driven by an Elasticsearch query.

6 Monitoring Dashboards

Kibana visualizations are used to build dashboards for each business system, showing component health and host status. All Kibana configurations (saved objects) are version‑controlled and automatically imported during deployment, achieving Infrastructure as Code.

7 Extending the Monitoring Scope

New services can be added by updating Heartbeat configurations for health checks or extending Filebeat paths for additional log files. Application‑level metrics can be sent via StatsD to Metricbeat, though current Metricbeat lacks tag support.

Future work includes integrating Elasticsearch APM for tracing.

8 Conclusion

For private deployments, the emphasis is on an out‑of‑the‑box, feature‑rich solution rather than high scalability. Elasticsearch 7.6.0 and Kibana meet these needs, and with standardized deployment scripts and pre‑prepared configurations, a complete monitoring system can be stood up in under half an hour.