Server Monitoring Solution: Requirements, Design Decisions, and Implementation Details

Background: Monitoring is essential for any server‑side application to detect bugs, performance bottlenecks, and failures; it also supports capacity planning, alerting, and automated operations.

Requirements include functional availability monitoring, performance tuning (e.g., slow SQL, cache hit rate), real‑time alerts, fault diagnosis assistance, capacity planning, and automated actions such as auto‑scaling or service degradation.

Monitoring Objects cover OS, network, application (RD), URI, Spring, JDBC, JVM, security, JS, and baseline metrics, with detailed items for Spring (class, method, call count, total/average time, max concurrency, slowest, error count), URI statistics, data source pool metrics, JDBC statistics, exception details, and JVM health.

Design Decision : Adopt a centralized, metrics‑based monitoring platform without an agent (thin client model). Two approaches were compared: per‑application self‑monitoring (e.g., Druid) versus unified reporting to a monitor center.

Business Monitoring Process

1. Instrument business code with monitoring points. 2. Report collected metrics to the monitor center (or let the center pull them). 3. Provide visual dashboards for query and analysis. 4. Trigger alerts when configured thresholds are breached.

System Modules

1. Client – offers metrics definitions (counter, timer, etc.), AOP or annotation‑based instrumentation, a reporter component, and optional local buffering when the monitor center is unavailable.

2. MonitorCenter – receives metric packets (preferably UDP), processes events via a pipeline of handlers (metrics, storage, analyzer, notifier), manages caching/storage, schedules tasks, and supplies visualization and configuration interfaces.

Reporting API Interfaces

Counter interface:

void increment(String key);

void increment(String key, Integer delta);

Gauge interface: void addGauge(String key, Double value); Metric interface (supports distribution statistics): void addMetric(String key, T value); Convenient time metric: void addTimeMetric(String key, long timeInMillis); Log interface: void log(LoggerLevel level, String key, String message); All reporting is designed to use UDP for low overhead.

Key Issues

1. Key collisions – recommend prefixing keys with package or application name and using tags (e.g., host, port) to differentiate instances while still allowing aggregation.

2. Metrics data format – aligns with Google Cloud Monitoring and OpenTSDB, using key, timestamp, value, and one or more tags; supports various value types (bool, double, int64, string, etc.) and metric types (cumulative, delta, gauge).

3. Server failure handling – initially drop data, later implement local buffering for counters and other metrics.

4. Data storage – typical choices are time‑series databases such as RRD/rrdtool, Graphite/whisper, InfluxDB, OpenTSDB (on HBase).

5. High‑performance network ingestion and multi‑dimensional visualization are also addressed.

Supplementary Notes

URI‑based HTTP testing is better handled by dedicated testing tools; unit testing, CI (Jenkins) and code quality (Sonar) are recommended; load‑testing tools can be used for realistic performance evaluation.