Why Post-Mortem Action Items Fail at 10M QPS: Building a Closed-Loop Improvement System

Why "Recorded" Often Means "Not Fixed"

After an 18-minute core-path outage, a blameless post-mortem produced 12 action items: add validation, add alerts, improve runbooks, reduce permissions, schedule drills. Six weeks later, the same misconfiguration reached production. On-call still relied on user complaints; the updated runbook was missing. The post-mortem document showed 4 items "in progress" and 8 with no status. Teams assumed others would act; one alert was submitted but closed due to noise.

The problem isn't lack of records — records were complete. What's missing is a mechanism that continuously answers: who owns the outcome, when is it due, how is delivery verified, what interim measures hold the risk before completion, and how do we prove the fix actually lowered failure probability after closure.

Writing an improvement item into a document only creates risk debt; tracking it until effect is verified pays that debt.

In small systems, personal relationships and memory can sustain follow-through. At 10M QPS, failures span multiple services, teams, and infrastructure layers; a single post-mortem may generate dozens of action items. Any high-risk item that falls through the cracks lets the same trigger penetrate defenses again. The improvement closed-loop therefore becomes a production process requiring state, evidence, rhythm, and governance rules — not just post-meeting nudging.

Four Missing Constraints in Record-Only Processes

Action items typically degrade through a familiar pattern: immediate consensus on "must fix," then business priorities reclaim capacity, owners rotate or dependencies stall, and months later the items read "improve monitoring" or "optimize process" with no link to the original failure mode.

Every individual may act responsibly, yet record-only processes lack four constraints:

Decoupled from risk: Items describe "what to do" without linking to specific root cause, impact scope, or failure scenario. When business priorities shift, they become unquantifiable technical debt, naturally ranked below revenue-generating work.

Vague definition of done: "Code merged," "config deployed," "doc written" can all be marked complete, but they only prove activity occurred — not that the defense works. An alert that never fires in the target scenario is a premature victory.

Blurred ownership: Multiple owners often means no one owns the outcome. The service team waits for platform capability; the platform team waits for refined requirements; the post-mortem lead assumes the ticket system auto-progresses. Each makes locally rational choices while the risk stands still.

No clock on the process: Items lack explicit deadlines, escalation rules, and regular checkpoints. They don't surface blockers proactively, nor does urgency rise as deadlines approach — they simply age silently.

Therefore, the management object must shift: teams face a set of risks exposed by the incident but not yet eliminated; tickets are merely vehicles for those risks. Tasks can slip, but risk cannot pretend to disappear during a slip.

What an Executable Improvement Item Looks Like

"Add cache alert" looks like an action item but is merely a wish. It doesn't specify which signal to monitor, which failure mode it covers, who acts on the alert, or what constitutes completion. Even a diligent engineer might deliver a dashboard unrelated to the post-mortem goal.

An executable improvement item requires at least eight fields (illustrated in the article's diagram):

Risk description (linked to root cause, impact, failure scenario)

Permanent fix description

Interim mitigation (while permanent fix is pending)

Single owner (one person, not a team)

Collaborators

Target date

Acceptance criteria (written before development)

Evidence of acceptance (delivery, behavior, outcome)

Each field constrains goal, ownership, time, or evidence; omitting any creates an escape hatch.

The single owner is critical. They don't have to do all the work, but they continuously own the risk: decompose dependencies, fight for scheduling, surface blockers, organize acceptance. Collaborators can be many, but final state has one interpreter. Writing "Team X" as owner fails because a team cannot receive reminders or answer specific dates in a review.

Acceptance criteria must be written before development, not after. Writing them first forces abstract wishes into observable results. Example: "Improve degradation capability" becomes "When recommendation service latency exceeds 500 ms for 60 consecutive seconds, gateway automatically falls back to static results; core order API error rate stays below 0.1%; quarterly drill reproduces the scenario and saves the curves." Now engineering boundaries, trigger conditions, and success standards are clear.

Good action items describe evidence of risk disappearance, not just engineer activity.

Interim mitigations are not optional. Permanent fixes may take a month, but risk exists from the moment the post-mortem ends. During the wait, teams can add two-person review, freeze high-risk changes, tighten traffic thresholds, or place rollback scripts in the on-call runbook. Interim measures are rarely perfect, but they explicitly acknowledge the risk remains and add a reversible layer of protection.

Prioritize Risk, Not Development Effort

A severe incident often yields 10–30+ improvement items. Marking all as highest priority renders priority meaningless. What needs ordering is risk, not the emotional intensity of the meeting.

A practical assessment framework uses four factors: recurrence likelihood, impact if it recurs, current detectability, and exposure time before fix. The first three determine risk magnitude; the last determines how long we must endure it. Teams need not worship a precise formula, but must record the rationale so different post-mortems use the same yardstick.

Items can be grouped into three tiers:

P0: Directly blocks recurrence of the same severe failure, or repairs the last failed defense. Schedule immediately; if permanent fix can't be quick, deploy interim protection first.

P1: Reduces blast radius, speeds detection or recovery. Fits into current sprint.

P2: Improves long-term engineering quality. Goes into quarterly governance; doesn't share the same delivery cycle as stop-the-bleeding items.

The most common prioritization mistake is favoring "add monitoring" and "add docs." They're low dev cost and easy to close, so the list turns green fast. But monitoring only improves detectability; docs only aid human judgment. Neither replaces a hard guardrail that prevents the trigger. If an illegal config caused the outage, the highest-value fix is making illegal configs impossible to deploy — not alerting faster after they're live.

Use defense layers to audit the improvement mix: front line eliminates the trigger; next limits propagation; next enables fast detection and auto-recovery; last is manual runbooks. If a post-mortem only produces end-of-chain measures, the team is still thinking "how to rescue next time" instead of "why do we still allow it to happen."

The further forward the defense, the closer to "accident doesn't happen"; the further back, the more it relies on someone doing the right thing after failure occurs.

Manage Risk with a State Machine, Not "Percent Complete"

Once in the tracking system, items need clear states. Simple "open/closed" hides blockers, pending acceptance, and observation periods. Writing "80%" helps nothing — the last 20% may stall in another team for three months.

A state machine fits better (shown in the article's diagram):

Pending Confirmation: Not a default junk bucket. Owner and date must be confirmed shortly after post-mortem; otherwise auto-escalate to both leads.

Confirmed: Owner, date, acceptance criteria locked.

In Progress: Active work underway.

Blocked: Not a failure state — its value is surfacing the real problem: missing API, unapproved budget, unsupported platform version. With blocker reason, unblock condition, and next check date, managers can make resource decisions.

Pending Acceptance: Function works per design.

Observing: Running in production without new issues, and target metrics moving.

Closed: Observation period passed, no recurrence or side effects.

Cancelled: Not a silent delete. Requires decision-maker and reason: system decommissioned, risk covered by another item, or business accepts residual risk. Cancellation is a risk acceptance, auditable like closure.

State transitions must be evidence-driven. Code submit moves to Pending Acceptance, not Closed. Drill report moves to Observing. Only after the pre-agreed observation window with no recurrence or side effects can it close. Every rollback must record reason; these rollbacks expose whether acceptance criteria were too loose or dependency management weak.

Tracking Rhythm Determines How Long Items Survive

Closed-loop isn't daily nagging. Frequent private messages create pressure but don't resolve cross-team priority conflicts. Better: establish layered rhythms so each level handles only what it can decide.

Owners continuously update status and evidence.

Weekly: post-mortem lead reviews near-due, blocked, and pending-acceptance items.

Monthly: service leads review overdue items and risk acceptances.

Quarterly: examine root-cause clusters, recurrence rates, platform-level improvements.

Automated reminders handle "forgot," not "can't do." If the same item slips twice, more emails won't help — escalate to a resource or scope decision: split the permanent fix, extend interim protection, change owner, or explicitly accept risk.

Escalation mechanisms hand problems the individual cannot solve to people with decision authority.

Deadlines aren't arbitrary promises. P0 dates are set by risk window; P2 dates by planning cadence. For cross-team dependencies, the owner should embed milestones in the same item to avoid discovering at the final date that the first dependency never started.

An often-overlooked role: the post-mortem lead. The incident commander may not continue, but every post-mortem needs someone accountable for the completeness of the entire action-item set. They don't replace individual owners; they check for unmapped root causes, high-risk items that lost interim protection, and whether all closed items have evidence.

Acceptance Means Replaying Failure Conditions, Not Just Watching Deploy

The hardest step is acceptance because "system won't fail in future" can't be proven directly. The feasible method: replay key conditions in a controlled environment.

If fixing a detection gap, inject the corresponding anomaly and measure time from anomaly to alert delivery.

If fixing an isolation gap, take down a partition and verify blast radius containment.

If fixing a recovery process, execute rollback in a drill window and record recovery time and manual steps.

If fixing a permission issue, attempt the same operation with unauthorized identity and confirm rejection with audit trail.

Acceptance requires three evidence types:

Delivery evidence: code, config, docs, change records — proves solution entered target environment.

Behavioral evidence: drill curves, logs, alert notifications, audit records — proves system took expected action in target scenario.

Outcome evidence: error-rate ceiling, recovery duration, blast radius — proves the action changed the risk outcome.

Only the first type proves "we did it"; all three together approach "it's effective."

Acceptance scenarios must come from the post-mortem timeline and root-cause tree, not an idealized test. If the original failure involved retry storms, cache cold-start, and cross-region latency simultaneously, testing a single error response doesn't cover the real combinatorial condition. When full reproduction isn't possible in test, split into multiple partial experiments and use capacity models or fault trees to document remaining assumptions.

For high-risk improvements, add "reverse acceptance": check what happens when the new guardrail itself fails. Auto-rollback depends on control plane — can we still manual rollback if control plane is down? Misconfigured rate limiter — does it drop all legitimate traffic? New alert rule — could it cause notification storms during widespread flapping? Improvements don't add reliability for free; they introduce new states and dependencies.

Observation period length depends on trigger frequency. Daily triggers: one week may suffice. Monthly settlement or promo-traffic triggers: must cross that cycle. For rare events, waiting for natural occurrence isn't controllable; scheduled drills are the only reliable verification.

Why Closed-Loop Is Harder at 10M QPS

In smaller systems, root cause, fix, and verification often live in one team. At 10M QPS, failure chains are longer: traffic spike → gateway retries → downstream thread-pool exhaustion → shared cache thrashing → control-plane rate-limiting delayed. Each segment belongs to a different team; any team fixing only its local piece may not cut the whole propagation chain.

Action items become dependency graphs, not lists. "Gateway per-tenant retry budget isolation" depends on unified identity labels; "cache per-business capacity protection" depends on traffic profiling. Three deadlines don't reveal that label delay blocks two defenses simultaneously. Platform must maintain explicit dependency edges, compute critical path, and sync downstream risk windows when upstream blocks.

Local acceptance ≠ end-to-end effectiveness. Single-service load test passes ≠ traffic through gateway, service mesh, shared dependencies still maintains isolation. High-severity fixes need full-chain drills: verify control signals propagate across layers, degradation order is correct, multiple protections don't amplify each other.

Concurrent risk volume rises. Thousands of machines, releases, dependencies produce daily anomalies. If every issue enters one backlog, teams drown in noise. Items must cluster by root-cause cluster: ten "connection pool exhaustion" incidents don't need ten platform fixes; one global governance item solves the common problem, each service keeps only onboarding and verification sub-items.

Org boundaries become reliability boundaries. A shared component used by dozens of businesses can't have its improvement priority set by one incident team. Requires service owners, platform leads, and reliability governance roles to jointly adjudicate: who funds platform refactor, who does business onboarding, who verifies cross-team results.

"Done" also layers: platform capability release closes only the platform dev sub-item, not the whole risk. Only when target businesses all onboard, key traffic coverage hits threshold, and full-chain drill passes can the parent action item close.

At 10M QPS, the closed-loop unit isn't a code commit — it's a cross-system defense line truly connected.

Metrics That Tell If Closed-Loop Works

On-time closure rate is intuitive but dangerous alone. Teams can slice tasks tiny, lower acceptance bars, and hit pretty numbers. Real measurement watches both process and outcome.

Process metrics (pipeline health): item documentation completeness, owner confirmation latency, P0 interim coverage, on-time acceptance rate, blocker duration, closure evidence completeness. They quickly expose where the process piles up.

Outcome metrics (risk reduction): same-root-cause recurrence rate, severe incident count, detection and recovery time, auto-protection coverage, drill success rate. They move slowly but decide whether the loop has real value.

Metrics should be sliced by priority, team, and root-cause cluster. Mixing P0 and P2 closure times is meaningless; mixing high-frequency small failures with rare large failures masks risk. Median shows typical efficiency; P90 or max blocker time reveals tail latency.

Guard against "closure-rate worship." When on-time closure becomes a hard KPI, teams defer hard items to next cycle, loosen acceptance criteria, or rename items to "research." Safer: treat metrics as diagnostic signals. A team's blocker time spikes — is platform scheduling insufficient? A root-cause cluster recurs — did local patching miss the systemic issue?

Leadership's real focus isn't how many greens on the list, but how many high-risk windows remain exposed and what measures protect them. Red items aren't necessarily management failure; long-untouched, unexplained greens are more suspicious.

From Docs and Tickets to a Risk Governance Platform

Closed-loop capability can be built incrementally; no need for a complex platform on day one.

Phase 1: Unified template + existing ticket system. Focus: execute eight fields, single owner, acceptance evidence; weekly manual review. Fits low item count, simple team boundaries.

Phase 2: Connect post-mortem doc, tickets, notifications. Action items auto-created from post-mortem; status syncs back; due/blocked escalate by rule. Reduces copy errors, keeps post-mortem conclusions aligned with execution facts.

Phase 3: Add risk view and dependency graph. Platform shows exposed risk windows by priority, linked to services, root-cause clusters, historical incidents, drill results. Managers see "which risk class remains uncontrolled" instead of scattered tasks.

Phase 4: Automate evidence collection. Release system attaches change records; monitoring attaches metric curves; drill platform backfills results; system prompts whether closure criteria met. Automation lowers maintenance cost, but final risk acceptance still needs human judgment.

Tooling evolution has a floor: don't let the platform swallow context. Compressing items into a few dropdowns for easier stats makes it impossible for a reader six months later to understand why a decision was made. Structured fields serve retrieval and governance; the facts, reasoning, and counterfactuals from the post-mortem must stay. Both are indispensable.

For teams just starting, the minimum viable closed-loop is surprisingly plain: every item has a single owner, a deadline, and acceptance evidence; weekly review of high-risk and blocked items; nothing closes without evidence. Running these three things consistently for three months usually delivers more value than buying a feature-rich platform nobody maintains.

Let Every Incident Leave One Fewer Unlocked Door

Post-mortem explains the past; improvement closed-loop decides whether the future is actually different. A polished report doesn't bridge the gap — risk identification, ownership confirmation, execution tracking, scenario acceptance, and production observation do.

Early stage: templates and weekly reviews build discipline. As cross-team dependencies grow, wire state, escalation, and evidence into the ticket system. At 10M QPS, govern items by root-cause cluster, manage cross-system defenses with dependency graphs, and confirm they truly connect via full-chain drills.

When a team can produce evidence that the same trigger next time will be blocked earlier, impact contained smaller, or recovery faster — that's the finish line of closed-loop.

Look back at your last post-mortem: of the items marked green, how many replayed the original failure conditions, and how many only completed dev activity? The gap between those two numbers is where the next round of reliability investment should start.