off prem

Google Cloud outage shows it’s still hard to understand hyperscalers’ real resilience regimes

Single datacenter and just three services taken down by ‘upstream’ power problem, while the rest of a zone and region kept humming

Google Cloud last week experienced an outage that analysts say demonstrates that not all promises of cloudy resilience are created equal.

Google’s incident report explained that three services – the VMware Engine (GCVE), NetApp Volumes, and Bare Metal Solutions (BMS) – experienced a 15-hour outage due to a cooling failure in its europe-west4-a zone.

The report includes the following detail: “The datacenter serving europe-west4-a for GCVE, BMS, and NetApp has experienced a power failure, which subsequently caused a cooling failure.”

The important detail there is that Google uses a discrete datacenter for those three services.

Another notable element of the incident report is the admission that “An electrical fault occurred on the utility grid upstream of the datacenter, disrupting the electrical distribution gear and cooling equipment.”

Google hasn’t explained how an upstream failure caused that disruption but did say it “proactively turned down workloads in order to protect customer data from any risks posed by running infrastructure in a high temperature environment.”

Whenever your correspondent talks to hyperscalers or datacenter operators about how they ensure resilience, they tell me about their use of multiple redundant pieces of energy infrastructure, plus on-site generation capabilities that can keep a datacenter powered for days if necessary.

We’ve asked Google if it had generators or other energy sources at this site, and if so, why it nonetheless had to turn down workloads. We’ve not received a response at the time of writing. Google told us its incident analysis “is currently ongoing” and promised to follow up once it is available.

Hidden dependencies

We also asked Google if it advertises the fact that some of its services are tied to a single datacenter, a matter of interest because like other hyperscalers it divides its cloud into “regions” that typically comprise multiple “zones” spread across a city or other locale. Like its hyperscale peers, Google recommends placing workloads across different zones and regions to ensure resilience. Yet this incident shows some services can be tied to a single datacenter in a zone – and that those single datacenters can experience problems while the rest of the zone keeps working.

Analysts told The Register the outage shows organizations need to dig into clouds’ promises of resilience.

“The real issue is transparency: customers are generally told to use multiple zones and regions for resilience but are rarely given visibility into whether a particular managed service has a single-datacenter dependency within a zone,” said Biswajeet Mahapatra, principal analyst at Forrester. “As a result, many organizations assume the cloud abstraction provides more facility-level redundancy than may actually exist for specialized services."

“The underlying architecture is not necessarily unusual,” he added. “AWS, Azure, and Google all operate services that rely on dedicated hardware, storage platforms, or tightly coupled infrastructure that may not be distributed across multiple facilities in the same way as core compute and storage services.”

Gartner Director Analyst Adrian Wong reminded The Register of the 2023 outage at Google Cloud’s europe-west9-a region, the cause of which was a water leak that Google said “originated in a non-Google portion of the facility.”

Google uses a tool called “Spanner” to replicate data across zones, but in the flooded zone Google’s Spanner configuration didn’t work once one building became unavailable.

“It is very hard to figure out how an individual region is architected,” Wong said. “Our customers are often surprised by that,” he added.

The incident report for last week’s outage includes an apology.

“We know how much you rely on Google Cloud, and we regret the impact on your productivity,” the document states, before promising a final incident report will detail “preventative actions.”

But as this incident shows, knowing how Google plans to avoid future incidents of this sort won’t arm customers with the knowledge to understand if those mitigations will address hidden design issues that can reduce resilience. ®