Google Cloud’s so-called uninterruptible power supplies caused a six-hour interruption

Happened at a non Google data center near me years ago. Even though the PDUs were powered by the battery matrix all the time, the sense on the switchover relays from utility to generator were miswired after some critical hardware maintenance was done.

At the next generator test, when the generator shut down at the end of the test the battery is refused to switch back to utility.

It was a really big data center, and a really big power feed. There was no easy way to manually fix it because the contactors were so big that it was an arc flash danger. They lost a good portion of a day getting a power engineer on site to confirm the state of everything and get them reconnected.