How Big Data Can Revolutionize Operations Monitoring

Operations Monitoring Landscape

Many companies monitor only basic infrastructure (CPU, memory, etc.) with tools like Zabbix, while business‑level monitoring is fragmented, relying on ad‑hoc scripts or disparate custom tools, or third‑party platforms.

Key Data Sources

All monitoring data ultimately originates from logs, whether text or binary. Logs enable:

System health monitoring

Root‑cause analysis

Performance bottleneck diagnosis

Security incident tracking

What Can Be Extracted from Logs?

The primary output is metrics, which can be classified into:

Business layer (e.g., transactions per second, order creation rate)

Application layer (error counts, request latency, 95th percentile)

System resources (CPU, memory, disk, process health)

Network layer (packet loss, ping latency, TCP connections)

Unified Implementation Approach

Instead of solving each problem individually, build a platform that lets teams address any monitoring need.

Architecture overview:

The platform uses log‑collection agents (Flume, Storm or Spark Streaming) to filter, format, and store logs, then performs real‑time calculations and writes metric data to HBase.

Only two development tasks are needed:

One‑off creation of a dashboard that reads from HBase (e.g., using ELK or a custom UI).

Long‑term development of Spark Streaming jobs that define log‑processing logic and generate metrics.

Implemented dashboards include:

Status‑code heatmap showing top URLs with HTTP 500 errors.

Response‑time heatmap ranking URLs by average latency.

Trace system (similar to Google Dapper or Taobao EagleEye) that tracks request chains via a unique UUID.

Benefits of This Architecture

Minimal custom code: existing big‑data components handle collection, storage, and processing.

Excellent scalability: each component runs in a cluster and can be expanded horizontally.

Focused development: engineers only need to implement log analysis and metric extraction.

Big‑Data Thinking for Operations

Identify data sources (primarily logs).

Define what analyses and metrics can be derived.

Select appropriate big‑data components to assemble a modular solution.

Key considerations include standardizing log formats across complex product lines and leveraging Spark SQL for ad‑hoc multi‑dimensional analysis when a data‑team is available.

Final Thoughts

The author built the first architecture in just over a day, with additional days spent on trace analysis and UI visualization, demonstrating that a robust, big‑data‑driven monitoring platform can be assembled quickly using off‑the‑shelf components.