On October 26 we covered the power failure that took out Google Cloud's europe-west2-b zone in London, based on the preliminary report. On November 4, Google Cloud posted its final incident report on the status page, with a root cause far more specific than the preliminary one. According to the final report, the main impact was resolved within 6 hours and 8 minutes (the preliminary report said 6 hours 10 minutes).

The Root Cause: a Maintenance Posture and Where a Sensor Sat

  • The facility uses a 2N utility design: two independent utility feeds, A and B, backed by a bank of high-voltage generators managed by a dual-redundant Programmable Logic Controller (PLC). The UPS batteries provide roughly five minutes, enough for the PLC to detect undervoltage, start and synchronise the diesel generators and transfer the load.
  • A quarterly generator load test on October 14 found issues, and to repair feed B its breaker was opened, leaving the facility in a temporary single-feed (1N) maintenance posture with feed A carrying the whole critical load.
  • At 22:53 Pacific on October 19 (06:53 BST on the 20th), feed A failed because of an underground cable fault on the utility network outside the facility. The PLC's undervoltage sensing was wired upstream of the feed breakers: feed B's breaker was open, but the line above it was still live, so both the active and standby PLCs, running identical logic, saw "a healthy utility source" and never commanded the generators to start. When the on-site team tried a remote load transfer through the PLC, the same reading blocked it.
  • Judging that the UPS would run out before remote operations could be sorted out, the team moved the facility to the standby utility source, but the batteries were depleted during the transfer and the data halls lost power.

Why Recovery Took Six Hours

  • All times BST: impact began at 07:00; the standby utility supply was restored at 07:54, the data halls had power again at 08:20, clusters began recovering at 08:40 and network devices were back at 08:45.
  • Storage was the slow part: the sudden power loss left a large number of storage devices with unclean shutdowns, which began recovering at 08:50, and responders deliberately held back services that would have made it worse. With so many devices this took a long time; storage was sufficiently restored only at 11:13, and traffic was gradually re-admitted from 11:45.
  • Google Cloud services and APIs were fully available at 12:11 (some GCE virtual machines were still not back), core platform disruption ended at 12:29, at 13:08 99.5% of GCE virtual machines were healthy and new ones could be created, and the closure notice went out at 15:03.
  • The other two zones suffered too: cross-zone calls to the offline zone timed out, and the load redirected by failover saturated compute capacity in the remaining zones, producing intermittent 500 and 503 errors across the region.

Google's Fixes, and Lessons for Your Own Servers

Google's action items include: the PLC was reconfigured and validated on October 20 and again on October 26, when a generator transfer test under the full critical load was also completed; the utility located the cable fault on October 26, expects to repair feed A by November 13, and by November 17 the facility should be back on feeds A and B with another generator load test; and a full review of the PLC code is due by November 6. For our own servers, three points are worth keeping: redundancy quietly becomes a single point of failure during maintenance, because while one machine is being serviced the other is the only one; monitoring should check the real state, not one upstream signal; and after a sudden power loss the slowest part is often storage recovery, so keep a copy of your data elsewhere using one of these backup strategies. For the thinking behind multi-region setups see why one node isn't enough, and for choosing locations see picking a low-latency data center node.