AWS US-EAST-1 Outage: DynamoDB DNS Race Condition Triggers EC2 Congestion Collapse

This article analyzes the October 19–20, 2025 AWS US-EAST-1 outage, which began with a DynamoDB DNS resolution failure and cascaded into a 12-hour EC2 congestion collapse and an NLB health-check failover.

Timeline

19 Oct 2025 23:48 PDT – Initial failure: AWS detected a significant error rate increase in the DynamoDB API in US-EAST-1.

20 Oct 2025 00:26 PDT – Root cause preliminarily identified: AWS announcement [1] stated the cause was a DynamoDB regional endpoint DNS resolution issue.

20 Oct 2025 02:24–02:25 PDT – Core service recovery: AWS reported the DynamoDB DNS issue mitigated at 02:24 PDT, including DNS record fixes.

20 Oct 2025 within hours – Cascading service recovery: Downstream services (EC2 instance launches, NLB, Lambda) recovered slowly despite DynamoDB restoration.

20 Oct 2025 15:01 PDT – Full recovery announced: All AWS services in US-EAST-1 returned to normal.

Even in hindsight, AWS engineers responded well; a detailed post-incident summary [2] was published three days later, offering rare transparency.

Root Cause Analysis: DynamoDB DNS Race Condition

The incident was triggered by a latent defect in DynamoDB's automated DNS management system, causing endpoint resolution failures for dynamodb.us-east-1.amazonaws.com.

The author illustrates the automated DNS management system:

For resiliency, the DNS Enactor operates redundantly and independently in three Availability Zones (AZs). Each instance looks for new plans and updates Route 53 by replacing the current plan with a new plan using a Route 53 transaction, ensuring consistent endpoint updates even when multiple Enactors attempt concurrent updates.

This race condition was anticipated, but the design assumed eventual consistency.

The author reconstructs how the eventual consistency assumption broke:

AZ1's DNS Enactor fetched DNS plan 41, found it newer than the last executed plan 40, prepared to update Route 53, but stalled for unknown reasons.

The DNS Planner continued generating new DNS plans.

AZ2's DNS Enactor executed quickly and updated Route 53 with plan 43.

AZ1's DNS Enactor recovered; having already performed the check, it directly updated Route 53 with plan 41.

AZ2's DNS Enactor began a cleanup process, removing old plans prior to plan 43.

At this point, the DNS records for dynamodb.us-east-1.amazonaws.com were cleared. Because the active plan 41 record was deleted, the system entered an inconsistent state, preventing any subsequent DNS Enactor from applying plan updates.

The post-incident summary describes the anomaly clearly, but the author questions why the DNS Enactor deletes Route 53 DNS records during cleanup—whether due to a Route 53 limitation or over-engineering.

Mitigation

AWS has disabled the DynamoDB DNS Planner and DNS Enactor automation worldwide. Before re-enabling, they will fix the race condition and add protections to prevent incorrect DNS plan application.

EC2 Congestion Collapse

From a site reliability engineering perspective, the focus shifts to the congestion collapse triggered by the DynamoDB failure, referencing The Evolution of SRE at Google [3]: "Instead of asking 'What software service failed?' we ask 'What interactions between parts of the system were inadequately controlled?'"

DynamoDB access recovered, but the situation worsened: EC2 failed and remained impaired for 12 hours, far longer than the 2-hour DynamoDB outage.

The post-incident summary reveals two EC2 subsystems:

Droplet Workflow Manager (DWFM) – manages all underlying physical servers (called "droplets") that host EC2 instances.

Network Manager – manages network state for all EC2 instances and network devices, syncing state to relevant devices.

DWFM tracks each droplet's lease to know if a server is occupied and when it can be assigned to a customer. DWFM performs health checks on servers every few minutes, storing results in DynamoDB. During the DynamoDB outage, DWFM lost lease connectivity with droplets, preventing creation and state changes of new instances (running instances were unaffected).

When DynamoDB recovered, DWFM faced a massive backlog of droplet lease establishment tasks. However, DWFM did not process the backlog in a healthy manner (even slowly); it entered a congestion collapse state.

Engineers manually restarted DWFM to clear the task queue, allowing droplet leases to succeed, and applied request throttling to reach a metastable state.

EC2 instance creation resumed, but the Network Manager then began experiencing congestion. Similarly, engineers reduced load to restore metastability.

The final step was to fully remove the request throttles placed on EC2 subsystems, then gradually relax throttling for a slow recovery.

Mitigation

AWS is building an additional test suite to augment existing scale testing, exercising the DWFM recovery workflow to identify future regressions. They will improve the throttling mechanism in EC2 data propagation systems to rate-limit incoming work based on waiting queue size, protecting the service during high load.

Key point: improve throttling mechanism , validate recovery mechanisms.

NLB Erroneous Feedback

The NLB load balancer's chain reaction is notable: the NLB health check system received erroneous feedback due to stale network configuration and incorrectly executed an AZ failover.

Mitigation

For NLB, AWS is adding a velocity control mechanism to limit the capacity a single NLB can remove when health check failures trigger AZ failover.

Key point: control erroneous feedback .

The health check system behaved exactly as designed for the inputs it received—it is a reliable component, but the unsafe interaction with the faulty environment caused the problem. 1. https://www.aboutamazon.com/news/aws/aws-service-disruptions-outage-update 2. https://aws.amazon.com/message/101925/ 3. https://www.usenix.org/publications/loginonline/evolution-sre-google