Day63 Architecture Practice: Complete a 25‑Point Case Study Under Time Pressure

Case: City Charging Service Platform Refactor

A municipal EV charging platform integrates 50 000 charging piles. Users query idle piles, reserve spots, start charging, pay, and receive e‑invoices via a mobile app. Operators can remotely view device status and push configurations.

Peak‑hour max 6000 queries/start‑charging requests per second
95% of core requests must finish within 2 seconds, error rate ≤0.1%
A failed service instance must recover core service within 30 seconds
A zone failure must recover core service within 5 minutes
Add a new charging‑pile protocol within 5 person‑days
Operators may manage only devices in their own region
Partners may access only data of devices they produce; all operations must be auditable

Material 1 – Performance Bottleneck

When traffic reaches 6000 QPS, monitoring shows:

Actual completed requests ≈ 2400 /s
P95 latency = 7 seconds
Error rate = 8%
App CPU ≈ 40%, memory ≈ 52%
Database CPU = 96%, I/O wait spikes
Connection pool fully occupied, avg wait 2 seconds
One charging‑pile status join query consumes 62% of total DB time

Scaling the query service from 4 to 10 instances did not improve throughput; database connection contention worsened.

Further inspection revealed:

Every request repeatedly reads basic pile info, rates, and latest status from the DB
SQL lacks appropriate composite indexes
After a successful start‑charging, the flow synchronously waits for SMS, statistics, and e‑invoice tasks
Query and start‑charging share the same resource pool
No rate‑limiting or degradation during peak

Load‑test shows a stateless query instance can stably handle at most 500 QPS while keeping utilization ≤70% and still tolerate a single‑instance failure.

Material 2 – Reliable Messaging

Order service writes the order to the DB, then publishes a "charging‑started" event. Billing, notification, and statistics services consume the event.

Process crashes while sending the event → billing never receives it
Consumer ACK timeout → event redelivered, causing duplicate billing
Offline piles reconnect and resend the same start event
Failed messages are retried endlessly; no dead‑letter, alert, or reconciliation

Material 3 – Protocol Evolution & Access Control

Different vendors use distinct message formats and commands. Core code contains many vendor‑specific branches:

if vendorA ...
else if vendorB ...
else if vendorC ...

Adding a new vendor forces changes in device‑access, order, and billing modules. Permission checks are limited to two roles (operator, partner). Operators can change region_id in URLs to manipulate other regions; partners can change vendor_id to read other vendors' data. Some devices still share long‑term static keys.

Answer Summary

Quality‑Attribute Mapping

95% of core requests within 2 s → Performance

Single‑instance recovery within 30 s → Availability

New protocol in 5 person‑days → Modifiability

Operator limited to own region → Security

Performance & Capacity

Bottleneck : Database path – CPU at 96%, I/O wait high, connection pool saturated (2 s wait), one join SQL consumes 62% of DB time, while app CPU/memory are modest.

Why scaling instances failed : Adding app instances creates more DB connections and concurrent SQL, intensifying contention on the already saturated database.

Targeted optimizations :

Optimize the status‑join SQL and add suitable composite indexes.

Cache static pile info, rates, and hot status with appropriate eviction policies.

Move SMS, statistics, and e‑invoice processing to asynchronous messaging.

Isolate non‑core queries (e.g., recommendation) from the start‑charging pool; apply rate‑limiting or degradation during peaks.

After optimization, evaluate read‑write splitting, sharding, or DB scaling; avoid blindly enlarging the connection pool.

Capacity calculation :

Instance capacity = 500 QPS (stable) → 70% utilization = 350 QPS per instance.
(N‑1) × 350 ≥ 6000 → N‑1 ≥ 17.14 → N‑1 ≥ 18 → N ≥ 19
=> At least 19 query‑service instances are required.

Reliable Messaging & Idempotency

Use the Outbox pattern: the order service writes the order and an outbox record in the same local DB transaction, guaranteeing atomic commit.

A background publisher reads the outbox, sends the event to the broker, and marks the record as sent only after broker acknowledgment. Failures trigger limited retries, exponential back‑off, dead‑letter queues, and alerting.

Assign a globally unique event ID. Consumers check this ID (via unique constraint, processing log, or state machine) to ensure idempotent handling, preventing duplicate billing from redelivered events or repeated device reports.

Periodic reconciliation between order, billing, and message tables detects and compensates missing entries.

Protocol Evolution & Access Control

Protocol isolation : Define a unified device‑protocol interface (connect, report‑status, start, stop, configure). Implement a vendor‑specific adapter that translates vendor messages to the unified model. Core order and billing depend only on the interface, not on vendor checks. Adding a new protocol requires only a new adapter and configuration, verified by contract tests and simulated devices.

Access control : Issue independent, revocable credentials for operators, partners, and devices; retire long‑term static keys. Secure communication with TLS/mTLS. Apply RBAC for base roles, then augment with ABAC/object‑level checks that validate the requester’s region, vendor, and device ownership, avoiding reliance on URL parameters like region_id or vendor_id. Enforce least‑privilege, audit all actions, and generate alerts for cross‑region or cross‑vendor access attempts.