Between 14:46 and 20:31 UTC on November 11 (22:46 that night to 04:31 the next morning, Beijing time), the AWS Tokyo Region (ap-northeast-1) had API errors, failed instance launches, and then connectivity problems for new instances and for instances whose network configuration had changed, mostly in a single Availability Zone, apne1-az1, for about 5 hours 45 minutes. AWS's event page lists 37 affected services, including Lightsail, ELB, NAT Gateway, RDS, ECS and EKS. AWS stressed repeatedly that instances that were already running were not affected: the machines kept running, and what broke were operations that create or change things.

The timeline in AWS's notices (UTC)

  • 15:40: first notice; elevated API error rates, with launch failures concentrated in apne1-az1.
  • 16:40: root cause identified. CloudWatch alarms for EC2 and EBS in the affected zone were moving into Insufficient Data, so alarms set to fire on that state could go off; the DescribeInstanceStatus API was also returning Insufficient Data, and AWS told customers to rely on their own monitoring. ECS and Fargate task launches were failing, ELB provisioning was delayed, and NAT Gateway create and disassociate workflows were failing.
  • 17:38: early signs of recovery; CloudWatch data reporting correctly again.
  • 18:54: a new problem: connectivity issues for newly created or modified instances in apne1-az1, affecting new instance launches, new cluster creation, Auto Scaling groups, gateways and VPC gateway endpoints; connectivity to existing resources and traffic between instances were not affected.
  • 19:54: AWS advised customers to launch new instances in the two healthy zones, apne1-az2 and apne1-az3, and not to make changes that would affect connectivity to existing instances in the affected zone, such as attaching or detaching network interfaces or gateways and changing VPC peering.
  • 21:11: resolved.

What the closing summary says, and what it leaves out

According to AWS's closing summary, the first impact window ran from 14:46 to 17:46 and showed up as API throttling or 5XX errors and failed launches; some RunInstances calls returned success, but the instances never started and went straight to TERMINATED. In the second window, after the APIs had recovered, newly launched instances and existing instances with modified network configuration could not connect, which AWS attributes to delays in an internal subsystem responsible for propagating EC2 instance networking changes, related to the underlying root cause; everything was resolved at 20:31. AWS recommends retrying failed requests. What the root cause actually was, the summary does not say, and as of November 18 AWS's post-event summaries page has no entry for this event; the latest is still the US East event of October 2025.

Lessons for your own servers

  • When a region has trouble, the first things to break are often operations such as launching machines and changing networks. If you rely on auto scaling, or plan to launch machines during an incident, your scripts need retries and a way to switch to another zone or region.
  • "No data" means neither "down" nor "fine". Handle the two states separately in your alerts, and keep monitoring that probes from outside, for example scheduled pings and page requests from another machine, as covered in testing latency and packet loss with ping and mtr.
  • Do not touch network configuration during an incident. AWS explicitly advised against attaching or detaching interfaces and changing peering this time; changing a healthy machine can disconnect it.
  • Keep a backup of important services in another region that you can start at any time, for example Tokyo as primary and Singapore as standby; see our Singapore node and VPS backup strategies, and for choosing nodes, picking a low-latency node.