Tech

News

Updated

AWS says a DNS race condition in DynamoDB caused the October 2025 outage

Key facts

AWS says the failure spread to EC2 and Network Load Balancer in the same Region, and the event ran from 11:48 PM PDT on Oct. 19 to 2:20 PM PDT on Oct. 20.

On Oct. 19, 2025, a latent race condition in the DynamoDB DNS management system left the service's us-east-1 regional endpoint with an incorrect, empty DNS record, and AWS says that failure triggered the disruption that spread through its N. Virginia (us-east-1) Region.

What happened

AWS says customers in the region could not connect to DynamoDB through its public endpoint from 11:48 PM PDT on Oct. 19, because the systems that needed the service hit DNS failures.

AWS describes its DNS system as two parts: a DNS Planner that writes plans of load balancers and weights, and a DNS Enactor that applies each plan to Amazon Route 53. AWS says an Enactor that was unusually delayed applied an older plan just as another process cleaned up old plans. The older plan overwrote the newer one, and the clean-up then deleted it, which was now the active plan, so all IP addresses for the regional endpoint were immediately removed. The check that should have stopped the older plan was out of date by then, and the system needed manual operator intervention to correct.

AWS also says the DynamoDB failure reached EC2. Its state checks depend on DynamoDB, so the leases that track physical servers timed out. After DynamoDB recovered at 2:25 AM PDT, engineers could not rebuild the leases fast enough, and at 4:14 AM they throttled incoming work and began selective restarts of the hosts that manage those leases. Full EC2 recovery came at 1:50 PM PDT on Oct. 20.

Delays in network changes for new EC2 instances then made Network Load Balancer health checks fail on and off, which removed capacity from DNS. Engineers disabled automatic health check failovers for the load balancer at 9:36 AM PDT and re-enabled them at 2:09 PM PDT. AWS also lists errors in Lambda, Amazon Connect, the AWS Support Center and IAM sign-in, and says the full list of affected services is in its event history.

The three periods of impact

AWS gives three distinct periods of impact to customer applications. All times are Pacific, as AWS states them.

ServiceWhat customers sawTimes, as AWS gives them
Amazon DynamoDBIncreased API error rates; customers and other services could not establish new connections11:48 PM Oct. 19 to 2:40 AM Oct. 20
Network Load BalancerIncreased connection errors for some load balancers5:30 AM to 2:09 PM Oct. 20
Amazon EC2New instance launches failed; some new instances had connectivity issues2:25 AM to 10:36 AM Oct. 20 for failed launches; connectivity issues resolved by 1:50 PM

What AWS says it is changing

What the record does not yet say

AWS's summary gives no date for turning the DynamoDB DNS automation back on, and says that will happen only after the race condition is fixed. AWS also says it will keep working through the details of the event across all of its services and look for more ways to avoid a similar impact (AWS summary).

Background

AWS lists this summary in its post-event summaries index under its title and the date Oct. 19, 2025. For the status codes that web servers send when a service is unavailable or does not respond in time, see what the HTTP 503 status code means and what the HTTP 504 status code means.

Have another question?