Day 60 Cloud‑Native Case Study: Service Governance, Reliable Messaging, and Observability

Overview

Day 60 case study focuses on a regional medical appointment platform that has been split into micro‑services and deployed on a container cluster. The study asks the learner to choose appropriate mechanisms for service governance, TCC/Saga, reliable messaging, Kubernetes, Service Mesh and observability, and to explain their benefits and costs.

Materials

Five material groups are presented:

Dynamic instance scaling and fault propagation.

Cross‑service consistency problems in the appointment flow.

Message loss, duplication and backlog.

Improper use of Kubernetes (session storage, premature traffic, health‑check only, CPU‑only scaling, secrets in images).

Scattered governance rules that make fault location hard.

Question 1 – Service Governance

Explain why hard‑coding IPs is wrong, list the protection mechanisms required for an internal call, and why five immediate retries can amplify a failure.

Analysis

Instances in a container environment can be created, removed, rebuilt or migrated at any time; therefore a fixed IP cannot reflect the current set of healthy instances. A registration‑and‑discovery mechanism (e.g., a service registry or Kubernetes Service + cluster DNS) provides a stable address and lets callers obtain a list of ready pods. Load‑balancing then selects a healthy instance.

Instance dynamic registration, caller dynamic discovery, only send requests to healthy instances.

Required protections for a call include service discovery, load‑balancing, timeout, limited retry, circuit‑breaker, bulkhead, rate‑limit/flow‑control and degradation.

Five immediate retries turn 1 000 req/s into up to 6 000 calls/s, creating a retry storm that exhausts thread pools and connection pools, spreading the failure to downstream services.

Correct retry design must be idempotent, limited in count and total time, use exponential back‑off with jitter, and be combined with timeout, circuit‑breaker, rate‑limit and bulkhead.

Question 2 – Distributed Transaction

Design a consistency solution for the appointment flow, describing the normal path, compensation steps, idempotency requirements and cost.

Analysis

The appointment involves three independent data stores (registration, slot‑service and payment) and an external payment gateway, so a single database transaction cannot cover the whole process. The core pattern is “reserve‑then‑confirm‑or‑release”.

Slot service provides a TCC‑style interface:

Reserve/Try: mark a slot as reserved and set an expiration time

Confirm: after successful payment, turn the reservation into a confirmed appointment

Release/Cancel: on payment failure, timeout or user cancellation, free the reserved slot

Saga orchestration coordinates the local transactions:

T1 Create pending appointment

T2 Reserve slot

T3 Initiate payment and wait for result

T4 On payment success → Confirm slot and mark appointment paid

Compensation steps:

Payment failure or timeout → Release slot and cancel appointment.

Unknown payment result → Query payment status before retrying; if still unknown, alert and possibly refund.

Payment succeeded but subsequent confirm failed → Retry confirm; if still failing, alert, reconcile and possibly refund.

Idempotency is enforced by a unique business order number for payment requests and by making Reserve/Confirm/Release operations idempotent. A state machine and timeout scanner handle late or duplicate callbacks.

The approach avoids long‑lasting locks, improves concurrency, but adds complexity in compensation logic, state tracking and monitoring.

Question 3 – Reliable Messaging

Explain how to keep business transaction and message sending consistent, handle duplicate consumption, backlog and ordering.

Analysis

When the appointment DB transaction commits and the process crashes before sending the “appointment succeeded” event, the business state is persisted while the downstream consumers never see the event. The Outbox pattern solves this gap by writing the business record and a pending event record in the same local transaction.

INSERT appointment record

INSERT outbox event record

A background publisher (or CDC) reads the outbox, sends the event to a broker (Kafka, RabbitMQ, …) and marks the record as sent. If sending fails, the publisher retries; CDC can also replay the event.

Consumers must be idempotent. Each event carries a unique event_id. Consumers store processed IDs or use a unique DB constraint so that a repeated delivery does not cause duplicate SMS or audit entries.

Backlog handling includes monitoring queue length, oldest‑message age, fixing the failing downstream service, scaling consumer pods, applying timeout and back‑off, and moving permanently failing messages to a dead‑letter queue.

Ordering for a single appointment is guaranteed by using appointment_id as the partition key and by checking version numbers or state‑machine transitions on the consumer side.

Question 4 – Kubernetes

Propose cloud‑native runtime improvements and differentiate readiness, liveness and scaling metrics.

Analysis

Containers should be stateless; session data belongs in a shared store or token. Persistent data, files and messages stay in external databases, object storage or message systems. Kubernetes Deployment maintains the desired replica count and performs rolling updates. Service and cluster DNS give a stable address; the Service performs load‑balancing.

Probes:

Startup probe – protects slow start‑up, prevents premature killing.

Readiness probe – determines whether the pod can receive traffic; unready pods are removed from Service endpoints.

Liveness probe – detects unrecoverable process failure and triggers a restart.

Scaling should consider business‑relevant signals instead of CPU alone: QPS, P95 latency, connection‑pool usage, message‑queue backlog, etc. Scaling policies define target thresholds, min/max replica counts, cooldown periods and resource requests/limits. Configuration and secrets must be externalised via ConfigMap/Secret or a dedicated secret‑management service.

Question 5 – Service Mesh & Observability

Describe why governance is inconsistent across 80 services, what Service Mesh does, how it differs from API gateway and Kubernetes, and design a fault‑location workflow.

Analysis

When each team implements its own timeout, retry, routing, mTLS and metrics logic, policies diverge and upgrades require per‑service changes. Service Mesh abstracts these cross‑cutting concerns into a sidecar proxy (data plane) and a control plane that distributes uniform rules.

Differences:

API gateway – external entry point, handles authentication, routing, rate‑limit, protocol translation.

Kubernetes – container orchestration, deployment, scaling, service discovery.

Service Mesh – internal service‑to‑service traffic, provides load‑balancing, timeout, retry, circuit‑breaker, mTLS, telemetry uniformly.

Observability stack:

Trace ID – propagates through gateway, services and mesh; enables end‑to‑end request path view.

Metrics – per‑service QPS, latency percentiles, error rate, thread‑pool usage, queue depth.

Logs – searchable by Trace ID to reveal exact error messages.

Fault location example: a user sees an 8 s delay. Traces show the payment service consumes 7.6 s. Metrics confirm high error rate and thread‑pool exhaustion in the payment service. Logs for the same Trace ID reveal a third‑party payment timeout and connection‑pool exhaustion. The mesh’s control plane can also show which proxies applied retries or circuit‑breakers.

Service Mesh adds resource overhead (sidecar containers), extra network hop and operational complexity; its benefit must be weighed against service count and governance needs.

Combined Answer (Exam‑Ready)

For dynamic instance changes, use service registration/discovery (or Kubernetes Service + cluster DNS) and load‑balancing; configure timeouts, limited exponential‑backoff retries, circuit‑breaker, bulkhead, rate‑limit and degradation to avoid retry storms. Explain why five immediate retries amplify load.

For cross‑service consistency, adopt Saga orchestration with a TCC‑style slot service (Reserve/Try, Confirm, Release). Use a unique business order number, make all steps idempotent, and define compensation logic for payment failure, timeout or unknown result.

To guarantee that a committed appointment also publishes an event, use the Outbox pattern: write the appointment and the outbox record in the same transaction, then let a publisher or CDC reliably send the message and retry on failure. Consumers must be idempotent via event_id or unique constraints; monitor backlog, expand consumers, use dead‑letter queues, and partition by appointment_id to preserve order.

Design services as stateless; store sessions in shared storage or tokens. Use Kubernetes Deployment for replica management, Startup/Readiness/Liveness probes for lifecycle control, and scale on business‑relevant metrics (QPS, latency, queue depth) rather than CPU alone. Externalise configuration and secrets via ConfigMap/Secret or a secret‑management service.

Deploy a Service Mesh to centralise internal traffic governance, mTLS and telemetry; keep API Gateway for external entry, Kubernetes for container orchestration, and Mesh for intra‑service policies. Use a unified Trace ID, metrics and logs to locate the 8‑second delay: trace shows the slow payment service, metrics reveal the overload, logs expose the third‑party timeout.

Scoring Checklist

Each question includes a checklist of required items (e.g., dynamic registration, health checks, timeout, limited retry with back‑off, circuit‑breaker, bulkhead, rate‑limit, degradation, Saga steps, TCC interface, Outbox atomicity, consumer idempotency, partition key, probe types, multi‑metric scaling, secret handling, mesh benefits and costs, observability workflow, etc.).