Performance & Reliability Case Study: Identify Bottlenecks, Plan Capacity, Design Fault Tolerance

Scenario and Objectives

A provincial internet‑hospital platform provides doctor‑schedule queries, appointment booking, online consultation, payment and report retrieval. The business sets the following quality targets:

Peak inbound traffic: 5,000 requests / second (QPS)

95% of schedule‑query and appointment requests must finish within 2 seconds

Error rate ≤ 0.1%

Annual availability ≥ 99.99%

Core‑service recovery after a single instance failure ≤ 30 seconds (RTO)

Core‑service recovery after an availability‑zone failure ≤ 5 minutes (RTO)

Maximum data loss ≤ 30 seconds (RPO)

Performance Bottleneck Identification (Material 1)

During a normal period the system handles about 800 QPS with a P95 latency of 500 ms. When the peak load reaches 5,000 QPS the observed metrics are:

Actual throughput ≈ 2,200 QPS
P95 latency ↑ to 8 seconds
Error rate = 10%
Application CPU ≈ 38%, memory ≈ 55%
Database CPU ≈ 94%, I/O wait noticeably high
Database connection pool 300 connections all occupied, average wait 2.4 seconds
One schedule‑related SQL consumes 65% of total DB execution time

Increasing the appointment service from 4 to 12 instances does not improve throughput, indicating the bottleneck lies downstream of the application layer.

Why Adding Application Instances Does Not Help

Each instance maintains its own database connection pool. Expanding from 4 to 12 instances raises the maximum concurrent connections from 400 to 1,200, but the database can only process a limited number of concurrent queries. The result is more SQL executions, higher lock contention, longer queueing for connections, and increased latency—exactly the symptoms observed.

Optimization Measures (Material 2)

SQL and Index Tuning

Check execution plan and slow‑query log
Create appropriate indexes for filter, join and sort columns
Remove unnecessary fields, complex joins and duplicate queries
Optimize pagination and data model according to access patterns
Inspect lock wait, I/O and statistics
Set connection‑pool limit based on DB capacity

Caching Hot Data

Cache hospital info, doctor profiles, upcoming schedule (no strict strong consistency required)
Pre‑warm cache, add random TTL jitter, protect hot keys, handle cache‑penetration, breakdown and avalanche

Static Content via CDN/Object Storage

Store report images and static assets in object storage
Serve them through a CDN to offload network, thread and disk pressure from the application servers

Asynchronous Processing of Non‑Core Tasks

Keep the main request path limited to:
  • Authentication & rule check
  • Slot reservation & decrement
  • Appointment record creation
  • Result return
Offload SMS sending, statistics, audit logging and report generation to a reliable message queue with outbox, idempotency, retry, dead‑letter handling and backlog monitoring.

Horizontal Scaling and Resource Isolation

Deploy stateless appointment service instances behind a load balancer
Separate core appointment traffic from recommendation, historical‑report traffic using different instance groups, thread pools or connection pools
Consider read‑replicas or read‑write splitting, but account for replication lag, stale reads and fail‑over complexity.

Rate Limiting, Queueing and Degradation

Entrance rate‑limit to block traffic beyond capacity
Short queues to smooth brief spikes (cannot accumulate indefinitely)
Graceful degradation of non‑critical features (recommendations, complex history queries)
Isolation compartments to prevent a single function from exhausting all threads or connections

Capacity Planning (Material 8)

Target peak: 6,000 QPS. A single instance can stably handle 600 QPS while meeting the latency requirement. Planning utilization is limited to 70%:

Planned capacity per instance = 600 × 0.70 = 420 QPS
Instances without fault tolerance = ceil(6000 / 420) = 15
To survive one instance failure while still meeting the target, deploy at least 16 instances.

This estimate assumes the database bottleneck is resolved; otherwise additional capacity must be added at the storage layer.

Reliability Analysis (Materials 3‑5)

Current deployment:

2 gateway instances (same switch)

2 appointment service instances on the same host and availability zone

Redis single node

Database single primary

All layers share a core network switch

Fault detection relies on manual log inspection (average 25 minutes to switch)

Daily full‑backup; asynchronous cross‑region replication lag up to 3 minutes

These constitute single points of failure and shared fault domains, violating the RTO/RPO goals.

RTO and RPO Definitions

RTO (Recovery Time Objective) – the maximum acceptable downtime after a failure (30 seconds for a single instance, 5 minutes for an AZ failure).

RPO (Recovery Point Objective) – the maximum acceptable data loss window (30 seconds of appointment data).

High‑Availability Design

Deploy gateways and appointment services as multiple stateless instances across different hosts, racks and availability zones. Use health checks, automatic removal, rapid restart, version rollback and traffic routing to achieve the 30‑second RTO.

Provide HA for Redis (sentinel or cluster) and the database (primary‑secondary or cluster) with synchronous or near‑synchronous replication to meet the 30‑second RPO. Reduce backup‑only recovery to sub‑minute by using continuous transaction logs and bounded‑asynchronous replication.

Ensure network components (load balancer, service discovery, DNS) are also redundant.

Reliability Calculations (Material 4)

Given component reliabilities (per task window):

Gateway instance reliability = 0.98 (2 instances in parallel)
Appointment service instance reliability = 0.97 (2 instances in parallel)
Database reliability = 0.99 (single instance)

Parallel reliability formula: R_parallel = 1 – (1‑R)^n

R_gateway = 1 – (1‑0.98)^2 = 0.9996
R_appointment = 1 – (1‑0.97)^2 = 0.9991

Series reliability (gateway → appointment → database):

R_current = R_gateway × R_appointment × 0.99 ≈ 0.9887 (≈ 98.87%)

If the database is upgraded to two independent nodes (each 0.99 reliability):

R_db_parallel = 1 – (1‑0.99)^2 = 0.9999
R_improved = 0.9996 × 0.9991 × 0.9999 ≈ 0.9986 (≈ 99.86%)

Annual availability target 99.99% allows only about 52.6 minutes of downtime per year (525,600 minutes × 0.0001).

Verification Plan (Material 5)

Performance Validation

Baseline test: measure single‑instance CPU, memory, DB query latency, slow‑SQL.

Load test: ramp to 5,000 QPS with realistic query/appointment/payment mix; verify P95 ≤ 2 s, error ≤ 0.1 %, throughput scales, and no resource saturation.

Stress test: increase load beyond 5,000 QPS to locate the capacity breakpoint; ensure rate‑limiting, queueing and degradation protect the core path.

Capacity test: simulate 6,000 QPS with 70 % utilization and one instance failure; confirm the system still meets latency and error targets.

Long‑duration stability test: run at high sustained load for several hours; watch for memory/connection leaks, cache‑hit stability, message backlog growth, and tail‑latency drift.

Reliability Validation

Terminate a single appointment instance – measure recovery time (must be ≤ 30 s).

Failover the primary database – verify automatic switch, data consistency and RPO ≤ 30 s.

Disable the Redis node – confirm HA fallback or graceful degradation.

Cut network connectivity for one availability zone – ensure traffic reroutes and service recovers within 5 minutes.

Inject a faulty software version across all instances – validate canary/gray‑release detection and rapid rollback.

Restore from backup to a fresh environment – compare data timestamps to ensure no more than 30 seconds of appointment data is lost.

Metrics to monitor during all tests include inbound QPS, completed throughput, P50/P95/P99 latencies, error rate, CPU/memory/GC, DB connection wait time, lock wait, I/O, cache hit ratio, message queue depth, fault‑detection latency, switch‑over time and data‑loss window.

Acceptance criteria are directly tied to the original goals: 5,000 QPS with P95 ≤ 2 s and error ≤ 0.1 %; 30‑second recovery for a single instance; 5‑minute recovery for an AZ failure; ≤ 30 seconds data loss; and overall annual availability ≥ 99.99% as demonstrated by the reliability calculations and test results.