Adaptive Service Monitoring and Self‑Healing Calls for Microservices

Service Runtime Monitoring

Microservice nodes expose built‑in counters for each HTTP endpoint (QPS, latency, error count, response codes). The platform uses an auto‑discovery + auto‑capture pipeline:

The HTTP framework registers a metric collector when the service starts.

During each request the collector updates the counters.

A background task periodically reads all counters, aggregates them and serialises the result as JSON.

The JSON payload is sent through a message queue to a dedicated monitoring‑storage node.

In addition to application‑level counters, JVM metrics are collected (heap usage per region, GC count & time, class‑loading stats, CPU utilisation, thread pool usage). All monitoring data share the same JSON schema, for example:

Service Context Monitoring

Context monitoring captures system‑level resources that directly affect service performance: CPU utilisation, memory consumption, open connections, per‑port traffic (KB), and disk usage.

Three acquisition methods are supported:

System commands – e.g. netstat, top, du, df.

System APIs – raw‑socket calls (C/Python) to obtain per‑port byte counters.

Procfs files – reading /proc/<pid>/... for process‑specific statistics.

The collected context data are sent by a lightweight heartbeat client to the central heartbeat service, stored alongside service‑registration information and kept time‑synchronised with it, enabling adaptive adjustments.

Adaptive Service Invocation

Load Balancing and Failover

The platform supports two balancing modes:

Proxy mode – a front‑end proxy (Nginx, HAProxy, etc.) forwards requests to service nodes.

Client mode – the client obtains the address list from a service registry (e.g., Zookeeper) and applies its own balancing algorithm.

Each service interface registers a balancing strategy and a failover policy, both of which can be updated at runtime.

Four failure categories are distinguished:

Unable to connect to the target address.

Call timeout (service alive but unresponsive).

Recoverable business exception – switching to another node may succeed.

Non‑recoverable business exception – switching cannot help.

Failover is performed for the first three categories. Two isolation strategies are defined:

Soft isolation – remove the failing address from the client’s discovery list.

Strong isolation – invoke a control API to stop the faulty service capability on the node.

Automatic Retry

Retry is triggered by the node that first performed failover. The node caches the failing address and, on a subsequent request thread, attempts the call again without allocating extra resources.

If the retry succeeds, the node lifts the isolation and notifies the registry.

If retries are exhausted without success, the registry may enforce strong isolation.

Isolation Mechanics

Isolation is applied only when the failed service does not affect data consistency. Soft isolation hides the address from discovery; strong isolation stops the service capability via a lifecycle‑management API. The isolation workflow is:

The registry receives a failure feedback and marks the corresponding address as unavailable in its cache.

Subsequent heartbeat synchronisations hide the address from other nodes.

If a node repeatedly fails after the configured retry limit, the registry issues a strong‑isolation command to the node’s control interface.

The node’s heartbeat client executes the command, stopping the faulty capability (or restarting it if the policy specifies a restart‑instead‑stop action).

Together, these mechanisms provide a resilient microservice computing platform that can monitor runtime and context metrics, adaptively balance load, fail over on errors, retry transient failures, and isolate unhealthy nodes without manual intervention.