Designing Scalable Monitoring with ELK and GPE: A Practical Guide

System Scale Overview

8 platforms

100+ servers

10+ cluster groups

600+ micro‑services

Millions of users

Key Monitoring Challenges

Visibility into container health and resource usage

Observability of thousands of micro‑service endpoints

Cluster‑level performance analysis and capacity planning

Management of large numbers of agent‑side configuration scripts

Observability Architecture

The solution combines a log‑centric stack (ELK) with a metric‑centric stack (GPE) to provide end‑to‑end observability. Alerting is routed through email, SMS, DingTalk and custom webhooks, with a 24/7 monitoring centre.

Log stack (ELK): Elasticsearch + Logstash + Kibana + Redis

Metric stack (GPE): Grafana + Prometheus + Exporter plugins + Consul for service discovery

ELK Log Stack

ELK provides reliable collection, storage and visualization of structured logs from distributed services.

Elasticsearch – distributed, REST‑ful search engine built on Lucene; handles automatic sharding, replication and zero‑configuration clustering.

Logstash – pipeline that ingests raw logs, applies filters, and forwards them to downstream stores.

Kibana – web UI for querying and visualizing data stored in Elasticsearch.

Redis – used as a buffering queue between Logstash shipper and Logstash indexer.

Typical workflow:

Logstash shipper monitors each service, parses logs and pushes them to Redis.

Logstash indexer reads from Redis, enriches the data and writes structured documents to Elasticsearch.

Critical (e.g., ERROR) logs trigger email or webhook alerts.

Kibana reads from Elasticsearch to render dashboards and enable ad‑hoc queries.

GPE Metric Stack

For low‑level system and application metrics, the GPE stack replaces ELK’s log‑only approach.

Grafana

Grafana is an out‑of‑the‑box visualization platform that supports multiple data sources, flexible dashboards and built‑in alerting.

Prometheus

Prometheus scrapes metrics via HTTP and stores them in a time‑series database.

Multi‑dimensional data model (metric name + key/value labels)

Powerful query language (PromQL)

Single‑node operation without external storage dependencies

Pull‑based collection (optional push‑gateway)

Service discovery or static configuration for target selection

Rich set of visualizations and dashboard templates

Consul

Consul provides dynamic service discovery, health checking, a hierarchical key‑value store and multi‑datacenter support, enabling exporters to register and deregister automatically.

Service discovery: clients locate APIs, databases, etc., via DNS or HTTP.

Health checks: monitor HTTP endpoints or node resources and expose status for routing decisions.

Key‑value store: store configuration, feature flags, leader election data, etc.

Multi‑datacenter: seamless operation across geographic regions.

GPE Workflow

Each Exporter registers its HTTP endpoint with Consul.

Prometheus queries Consul to obtain the current list of exporter targets.

Exporters collect system or application metrics (CPU, memory, GC, custom business KPIs) and expose them on /metrics.

Prometheus scrapes the /metrics endpoints at configured intervals and stores the data.

Grafana uses Prometheus as a data source to build real‑time dashboards.

Grafana’s alerting engine evaluates PromQL expressions and sends notifications via email, DingTalk or custom webhook.

Conclusion

The combined ELK + GPE architecture delivers a unified observability platform: ELK handles high‑volume, unstructured log data, while GPE provides low‑latency, dimensional metrics and alerting. By leveraging Consul for dynamic service discovery, the stack scales with micro‑service growth and reduces the operational burden of managing static configuration scripts.