Building an Effective Monitoring and Alerting System for Rapidly Evolving Services

Observability is the foundation for reliable debugging and diagnosis in fast‑changing business and architecture environments. Monitoring and alerting are a subset of observability, and the article explains how the team expands traditional ops‑only monitoring to a shared, organization‑wide practice.

Data sources are classified into component, infrastructure, and application layers. Examples include gateway access logs formatted as JSON, MySQL slow‑query logs (stored in a slow_log table), Redis slowlog queues, Elasticsearch task APIs, and TCP connection statistics obtained via ss -toempi. The article shows how each source can be treated as metrics, logs, or traces, and how they can be transformed between these forms.

Collection and storage uses a modern ELK‑Kafka‑Prometheus stack. Filebeat forwards raw logs to Logstash, which writes to Elasticsearch; Kafka decouples Filebeat from Logstash to avoid back‑pressure and high CPU usage. For long‑term metric storage, Prometheus data is replicated with Thanos on AWS S3, while ClickHouse handles massive log and trace volumes with distributed tables and index lifecycle management. The author discusses trade‑offs such as shard count, replica placement, and the impact of using distributed tables for both reads and writes.

Metric monitoring systems are evaluated: legacy tools like Cacti and Zabbix struggle with cloud‑native, thousands‑of‑node environments, whereas Prometheus offers pull‑based collection, service discovery for AWS, Kubernetes, and Consul, and a powerful time‑series engine. High availability is achieved by deploying multiple Prometheus instances across data centers that monitor each other.

Visualization and analysis combine Kibana, Grafana, and custom scripts. The article demonstrates a Timelion query that compares today’s access‑log volume with the same period last week, a PromQL expression for API latency, and a ClickHouse SQL query that aggregates recent call‑chain data with P50 latency. Each example includes the exact query syntax inside pre blocks.

Alert visualisation moves beyond plain text. Grafana’s static‑image API and Python’s Matplotlib are used to generate rich alert pictures that convey trends, spikes, and root‑cause paths, making alerts far more informative than a single metric threshold.

Challenges and future work include the lack of a single unified platform for all observability data, exploring BPF for kernel‑level metrics, adopting Flink for stream processing, and leveraging AI to spot anomalous patterns among thousands of time‑series. The article ends with an invitation for like‑minded engineers to join the effort.

log_format main '{
  "@timestamp": "$time_iso8601",
  "server_name": "$host",
  "clientip": "$remote_addr",
  "auth": "$remote_user",
  "request_method": "$request_method",
  "request_uri": "$uri",
  "args": "$args",
  "status": $status,
  "bytes": "$body_bytes_sent",
  "referer": "$http_referer",
  "agent": "$http_user_agent",
  "x_forwarded_for": "$http_x_forwarded_for",
  "responsetime": $request_time,
  "x_real_forwarded_for": "$http_x_real_forwarded_for",
  "upstreamtime": "$upstream_response_time",
  "upstreamhost": "$upstream_addr"
}'

CREATE TABLE `slow_log` (
  `start_time` timestamp(6) NOT NULL DEFAULT CURRENT_TIMESTAMP(6) ON UPDATE CURRENT_TIMESTAMP(6),
  `user_host` mediumtext NOT NULL,
  `query_time` time(6) NOT NULL,
  `lock_time` time(6) NOT NULL,
  `rows_sent` int(11) NOT NULL,
  `rows_examined` int(11) NOT NULL,
  `db` varchar(512) NOT NULL,
  `last_insert_id` int(11) NOT NULL,
  `insert_id` int(11) NOT NULL,
  `server_id` int(10) unsigned NOT NULL,
  `sql_text` mediumblob NOT NULL,
  `thread_id` bigint(21) unsigned NOT NULL
) ENGINE=CSV DEFAULT CHARSET=utf8 COMMENT='Slow log';

SELECT a.*, b.*, c.*, d.*, e.*
FROM information_schema.PROCESSLIST a
LEFT JOIN information_schema.INNODB_TRX b ON a.id = b.trx_mysql_thread_id
LEFT JOIN information_schema.INNODB_LOCKS c ON b.trx_id = c.lock_trx_id
LEFT JOIN information_schema.INNODB_LOCK_WAITS d ON c.lock_id = d.requesting_trx_id AND c.lock_id = d.requested_lock_id
LEFT JOIN information_schema.INNODB_TRX e ON d.blocking_trx_id = e.trx_id;

ss -toempi

kafka-consumer-groups.sh --bootstrap-server kafka:9092 --group consumergroup --describe

.es(index=log_access*, q='(awip.city:"福州市" AND (server_name:"j.test.com" OR server_name:"j2.test.com"))')
  .divide(.es(index=log_access*, offset=-1w, q='(awip.city:"福州市" AND (server_name:"j.test.com" OR server_name:"j2.test.com"))'))
  .subtract(1).label('福州市今日和上周的涨幅下跌情况(1,-1)').bars()

1 - sum by (interface, method) (increase(dubbo_api_total_invocation_times{interface="com.pupu.test.api.client.Test", method="sendTest(TestDTO)"}[1m] offset 1d)) /
    sum by (interface, method) (increase(dubbo_api_total_invocation_times{interface="com.pupu.test.api.client.Test", method="sendTest(TestDTO)"}[1m]))

SELECT arrayStringConcat(n, '->'), start, end, c, p50
FROM (
  SELECT n, start, end, count() AS c, quantile(0.9)(totalDuration) AS p50
  FROM (
    SELECT groupArray(tags.type) AS t,
           groupArray(name)[1] AS start,
           groupArray(name)[-1] AS end,
           arrayCompact(groupArray(localEndpoint.serviceName)) AS n,
           sum(duration) AS totalDuration
    FROM test_distributed
    WHERE (time_date = toDate(today())) AND (time_datetime > subtractMinutes(now(), 5))
    GROUP BY traceId
  )
  WHERE (t[1]) = 'webmvc'
  GROUP BY n, start, end
)
ORDER BY c DESC
LIMIT 10;

Overall, the article demonstrates a comprehensive, observability‑first approach to building a scalable monitoring and alerting system, detailing concrete configurations, code, and visualisation techniques while acknowledging current limitations and future research directions.