A 1968 tower collapse forced civil engineers to design for the failure of any single element. Distributed systems have the parts — circuit breakers, bulkheads, failovers — but not always the discipline.

This piece continues a habit I lean on: exploring other engineering disciplines to see what problems they solved before software hit the same wall. Last time I borrowed from urban planning in “

TL;DR: Cascading failures are progressive collapse. After the Ronan Point disaster in 1968, civil engineering introduced the rule "don't collapse out of proportion to the cause" as a mandatory building code, built on three named, escalating mechanisms: tie forces, alternate load paths, and key elements. Those three map almost one-to-one onto timeouts/circuit breakers, real failover, and hardened critical dependencies. The catch: civil engineering ships them as a design check you run on every element, sized to the severity of the failure. We often bolt them on only after an incident. But some failures burn users' trust, and those paths need to be solid before anything breaks. The artifact to steal is the check: Find the load path. Size the tie. Name the key element.

The shape of the collapse

On the morning of 16 May 1968, a resident on the 18th floor of a brand-new tower block in East London lit a match for the kettle. Gas had been leaking behind a cooker. The explosion wasn't huge. They survived it, knocked unconscious in a kitchen that stayed standing. But it blew out a load-bearing wall: one of the precast concrete panels holding up the four flats stacked above.

With the wall gone, the flats above had nothing under one corner. The slabs landed on the flats below, which had been fine until several floors of debris arrived. Those gave way too. One whole corner of a 22-story building unzipped, top to bottom. The corner held the living rooms, and at that early hour most residents were still in bed; four people died.

The building was Ronan Point. It had opened two months earlier.

Now lower the stakes from lives to revenue, and look at its shape. It's an outage. You've watched this one happen. A non-critical dependency gets slow: recommendations, a reports service, a sidecar. It doesn't crash, it just goes from 20ms to 2 seconds.

Upstream, the caller was written to wait. So it waits, its threads fill up with calls stuck on the slow thing. New requests show up and find nothing free. Now the caller is slow. Its caller starts waiting on it. A minute later, a feature you'd never page someone for has taken down checkout.

Tiny trigger, huge damage. That gap between how small the cause was and how big the collapse got is the whole subject of this article.

We have a pile of names for the pieces: cascading failure, retry storm, thundering herd, "we need a circuit breaker." What we don't have, as a field, is a single design discipline that ties the pieces together and tells you when each is required.

Civil engineering has had exactly that since 1970, written into the building code and legally required.

Progressive vs. disproportionate: two words worth stealing

Structural engineers split apart two things software lumps together.

Progressive collapse is the chain reaction. One element fails, the thing it held drops, the thing under that gets overloaded, and the failure spreads.

Disproportionate collapse is when that spread does damage way out of proportion to the trigger. A small, local, even expected failure takes down a huge chunk of the structure.

The first does not have to become the second.

A progressive collapse that gets contained (stopped after one floor, one bay) is survivable. It's expected, even. The whole job of the discipline is to stop progressive from turning into disproportionate.

Ronan Point's real legacy: collapse became a design input

Back to East London, because what the inquiry found is where the borrowing starts. A striking detail in the Ronan Point case - the design met the building code of the day.

Britain's official inquiry found the failure was baked into the design. The building was a stack of precast panels resting on and bolted to each other. Each panel held up the one above. There was no second path for the load. Pull out one panel — by explosion, by a truck hitting the base, by anything — and everything above it was, by design, holding onto nothing.

And the fix wasn't "stronger panels." It was a new idea.

Through the 5th Amendment to the Building Regulations in 1970 (now Part A, under the heading "Disproportionate Collapse"), the UK added a design requirement. Today's wording:

"The building shall be constructed so that in the event of an accident the building will not suffer collapse to an extent disproportionate to the cause."

Europe's structural code calls the same idea robustness:

"the ability of a structure to withstand events like fire, explosions, impact or the consequences of human error, without being damaged to an extent disproportionate to the original cause."

Read those like a spec. Because that's what they are.

They admit accidents will happen (explosions, impacts, human error) and they make containing the damage the design target. Failure is assumed. Disproportionate failure is banned. The thing you're designing is the blast radius.

That's the move. We design software for correctness and treat failure as an ops problem: runbooks, alerts, retries bolted on after the incident. Civil engineering promoted "what happens when a piece of this is gone" to a first-class design input. The failure case goes in the blueprint, not the postmortem.

But a principle isn't a method. The reason this is a borrowed architecture and not a borrowed slogan: civil engineering didn't stop at "don't collapse." It turned the principle into three named, escalating mechanisms.

Those three are the whole transfer. And to keep them honest, I'll run each one against a live dependency from the same worked example: an OrderService in the usual e-commerce shape, leaning on recommendations, inventory, and payments. One dependency per mechanism.

🔗 Mechanism 1: tie forces — the connection must not tear

Cheapest of the three is tie forces. Minimum required strength in the connections between elements.

Floors get tied to walls. Columns get tied vertically, foundation to roof. One European code even pins a number on it: an internal floor tie in a mid-consequence building has to hold roughly 80% of the weight of the floor strip it carries (its share of the dead load plus part of the live load), and never less than 75 kN.

What do ties actually buy you? Two things.

One is catenary action. Lose a column under a tied beam, and the beam can hang across the gap like a cable (a catenary), carrying its load sideways to the supports instead of just dropping. The load gets across the damaged area because the connection held.

The other the guidance literally calls holding columns in place. A sideways force shoves a column; if the beams framing into it have real tying strength, they keep it located, so it keeps carrying weight.

Note what a tie is not: a backup, or redundancy of any kind. A tie is the rule that the joint between two elements is strong enough that, when load suddenly shifts, the connection doesn't fail and drag its neighbor down with it.

It keeps the building acting like one connected thing instead of a stack of loose panels.

In software, the tie is the contract between two services under stress.

  • A timeoutis a tie.
  • A bounded retry(capped, with backoff, not infinite eager retries) is a tie.
  • Backpressure, making a fast producer slow down instead of burying a slow consumer, is a tie.
  • A circuit breakeris a tie.

None of these give you an alternative to the dependency. What they do is guarantee that when the dependency degrades, the connection degrades gracefully, instead of tearing the caller apart.

The slow-recommendations cascade I opened with? That's a missing tie. The connection had no defined strength under load, so the caller got dragged down too. The minimum tie force is the timeout you forgot to set.

So let's set it. Dependency one of our OrderService: recommendations, the same boring dependency that opened this article. It's disposable. Remove it and checkout still works; we show a static "popular items" list instead. All it needs is a cheap tie that prevents the opening cascade:

async function getRecommendations(userId: string): Promise<Product[]> { // Tie: tight timeout + circuit breaker so a slow dependency // can never pile up requests inside the caller. return recsBreaker.execute( () => withTimeout(recsClient.fetch(userId), 80 /* ms */), // Alternate path: a static fallback. Load reroutes to something // cheap. Blast radius = "slightly worse recs," not "no checkout." () => staticPopularItems(), ); }
(That static list is technically a second load path too — the most trivial one you'll ever ship. The next mechanism is for when the second path costs something real.)

🔀 Mechanism 2: alternate load path — remove it on paper and look

Mechanism two is the one with teeth: the alternate load path, checked by notional removal.

And it's exactly what it sounds like. An engineer takes each supporting element, one at a time, and deletes it on paper. Then checks: does the building still stand? Can the floors above bridge the gap? Can the floor below hold the debris if something does drop? And, critically, is the damage within an allowed limit?

In the UK guidance for a mid-upper-consequence building, after you remove an element the collapse must not exceed 15% of that floor's area, or 100 square metres, whichever is smaller, and must not spread past the floors immediately above and below.

That's a blast-radius budget, written into code, with a number on it. The structure isn't required to be unkillable. It's required to fail no bigger than this.

Our version of this is failover and redundancy. But the discipline is in two adjectives the civil version forces on you that we skip. Both are things we should look for in design reviews.

First: the failover has to actually re-route.

A replica that fails over to another replica, one that shares the same upstream dependency, the same config push, the same poisoned cache, hasn't rerouted anything. Same path, different color, and it dies in the same event. Notional removal is brutal about this. You delete the element and ask what carries the load now, for real, including everything the substitute also leans on. Most "redundant" architectures have never honestly answered that for their correlated failures.

Second: the damage limit has to be named up front.

This is the discipline we almost completely lack. I've almost never seen a team write down, before the incident, "the most this failure is allowed to take out is X." And choosing that number (one cell? one region? 15% of traffic? one tenant tier?) is most of the value. It forces you to admit which failures you're choosing to live with and which you're actually designing against.

Back to another example - inventory. Run the removal on the primary stock store and there is an alternate path, but it costs something: serve a cached or replica snapshot, accept bounded staleness. So we size the tie, and we name the blast-radius budget out loud ("we may oversell within a known, small window"), and the business signs off on it. On purpose, not by accident:

async function checkStock(sku: string): Promise<StockResult> { try { return await inventoryBreaker.execute( () => withTimeout(inventoryPrimary.read(sku), 150), ); } catch { // Alternate path that ACTUALLY re-routes: a different store with // an independent failure mode, not a replica sharing the primary's // fate. Blast-radius budget: bounded staleness, explicitly accepted. return inventoryCache.readWithStaleness(sku, { maxAgeMs: 5_000 }); } }
Both adjectives, earned: the cache has an independent failure mode, so it actually re-routes; and the budget has a name, so nobody discovers it during the incident.

And here's where software beats concrete. A building's notional removal is a calculation, done once, on a structure that then stands for decades. You can't go yank a real column out of an occupied tower to see what happens.

🧱 Mechanism 3: key elements — when there's no other path

Sometimes you run the removal check and the answer is ugly. You delete the element, the building comes down past the limit, and there's no practical way to give it an alternate path. A single transfer beam carrying a dozen columns: the element that, by its geometry, just has no understudy.

For these, the code has a third mechanism, and it's philosophically different: key element design.

That element gets formally labeled a "key element." Then it's designed to take a much higher, explicitly defined load: in the European code, an accidental design action of 34 kN/m² from any direction, on the member and everything attached to it.

Tie forces and alternate paths limit the spread after something fails. They assume the failure and contain it.

Key element design stops the element failing in the first place, because there's nothing to contain it with. The guidance says it straight: the key-element approach is "focused on preventing the supporting element being damaged," while the other two are "focused on limiting the spread of damage."

And you don't get to pick key-element design as your default. It's the fallback for when containment is impossible. You reach for hardening only after you've confirmed there's no alternate path. The code makes you try to design out the single point of failure before it lets you just make it really strong.

Every distributed system I've worked on has had key elements, whether or not anyone had named them.

  • The auth service everything checks and nothing routes around.
  • The primary database whose "failover" has never once been exercised under real load.
  • And dependency three of our OrderService:payments.

Remove payments and there's no honest alternate path. You can't "gracefully degrade" out of taking the customer's money.

The lesson is not "add redundancy until it stops being a key element." Sometimes you genuinely can't. It says instead to name the thing a key element and design it like one: isolate it so nothing else can hurt it, give it the highest reliability budget, the strictest SLO, the most redundancy you can buy, and most of your testing and attention.

For payments, that looks like this:

// Key element: no honest fallback exists, so the goal flips from // "re-route the load" to "fail rarely, and fail alone." // One narrow contract - the only door into payments. interface PaymentPort { charge(order: OrderId, amount: Money): Promise<Receipt>; } // The bulkhead: payments gets its own small, bounded pool. // At most 32 charges in flight and 8 waiting; request #41 is // rejected instantly instead of joining an invisible queue. // A payment slowdown can eat these 40 slots and nothing else - // browsing, cart, and the rest of checkout keep their threads. const paymentBulkhead = new Bulkhead({ maxConcurrent: 32, maxQueue: 8 }); async function takePayment(order: Order): Promise<Receipt> { // The tie, same as every dependency: bounded time, plus a // breaker so we stop hammering a provider that's already down. return paymentBulkhead.execute(() => paymentBreaker.execute(() => withTimeout(payments.charge(order.id, order.total), 2_000), ), ); } // What's deliberately missing: no catch, no fallback, no cache. // A failed charge surfaces as an honest error. That's the // key-element deal: spend the budget preventing this failure, // not hiding it.
You harden what you can't fail over.

⚖️ Consequence classes: proportionate, not uniform

If the three mechanisms were the whole story, the takeaway would be "do all three to everything." That's wrong and unaffordable.

The smartest thing civil engineering does is tell you how much robustness each part needs. And the answer is definitely not "the same everywhere."

Building codes sort structures into consequence classes by how bad a failure would be: people at risk, height, use, social cost.

  • A house up to four storeys: Class 1.
  • A block of flats, 5–15 storeys: Class 2B.
  • A stadium for 5,000+, or a building full of hazardous processes: Class 3.

And the required mechanisms escalate only with the class:

  • Class 1:normal construction. No extra robustness measures at all.
  • Class 2A:effective horizontal ties. That's it. The cheapest mechanism, just that one.
  • Class 2B:horizontal- andvertical ties,- orpass notional removal,- orkey-element design where removal fails.
  • Class 3:a full systematic risk assessment.

A house doesn't get a blast-radius analysis. A stadium doesn't get to skip one. Effort is proportionate to consequence, and that proportionality is mandatory.

Software's failure usually isn't missing the mechanisms. Most mature systems have circuit breakers and replicas somewhere. The failure is uniformity. We wrap everything in the same default breaker config, or we wrap nothing.

The right shape is the building code's shape, and I tier dependencies the same way: classify each one by the blast radius of its failure, then decide which mechanisms it needs. Our OrderService already walked the ladder:

  • Recommendations is Class 1.Most dependencies are. Let them fail; contain them with a cheap tie.
  • Inventory is Class 2B.A real alternate path and a named damage budget.
  • Payments is Class 3, and a key element.Two separate labels: the class comes from the consequence, the key-element label comes from having no alternate path. Risk assessment, isolation, reliability budget, your nights and weekends.

Three dependencies, three classes, three different answers, out of the same three questions. Spending Class 3 effort on a Class 1 dependency is how teams build elaborate failover for the recommendations widget while the single payments integration sits there unhardened.

Aviation tiers the same way and puts numbers on it. FAA rules tie the allowed probability of a failure to how bad it would be, from one in a thousand flight hours for a minor condition down to one in a billion for a catastrophic one. And a catastrophic condition doesn't get to come from a single failure at all.

The parts we have, the check we don't

None of this is alien to us. We independently invented every one of these mechanisms; the OrderService code above used them by name. The circuit breaker and the bulkhead are Michael Nygard's patterns from Release It! (the breaker is a tie; the bulkhead is a ship's watertight compartment, which ship engineers borrowed from long before we borrowed it from them). Cell-based architecture (partitioning a system into independent copies, each serving its own slice of traffic, so a failure stays locked inside one cell) is the alternate-path-and-blast-radius idea made real. A service mesh ships the ties as infrastructure instead of code re-written in every service.

What civil engineering adds is everything around the parts: the check run on every element, the damage budget written down before the incident, and the consequence class that says which elements deserve which mechanisms.

One piece of that inspection we're better equipped for than civil engineering itself — the removal check. A building's notional removal runs once, on paper. Chaos engineering (Chaos Monkey and everything after it) is the same check run against the live system, on purpose, over and over: remove a candidate, watch the load reroute or fail to, and stop guessing which element is secretly a key element.

Grid operators have run this check continuously for decades: North American reliability rules require a contingency assessment of the live grid at least every thirty minutes, and European law defines the N-1 criterion as the rule that whatever remains after a contingency can carry the new situation without breaching operating limits. But theirs stays a simulation, because you cannot de-energise a substation to check your model. So the ladder runs: a building gets the check once, on paper. A grid gets it every half hour, against a model of the live system. We can run it daily, on the real thing, because our worst case is a retry and theirs is a dark city.

So the missing piece was never another component. The inspection distills to three questions, asked of every dependency. They are the artifact of this piece.

The check: three questions

  • Find the load path.Notionally remove this dependency. Does the load reroute and the system survive within an acceptable blast radius? Yes → you have an alternate path, name its budget. No → go to question 3.
  • Size the tie.What's the minimum connection strength that stops this dependency's degradation from tearing the caller apart? Timeout, bounded retry, backpressure, circuit breaker.- Everydependency gets a tie.
  • Name the key element.No alternate path? Stop pretending. Label it a key element and give it the higher standard: isolation, reliability budget, strictest SLO, most testing.

That's the discipline. Not "add circuit breakers." The check.

And it doesn't stop at service-to-service calls. Everything in OrderService was a dependency you call, but the check works on anything that carries load — and it gets skipped most at the edges of the system.

Two design review examples

So let's finish the way I'd use this at work: a short design review of two edge components almost everyone owns and few teams audit. First, the API gateway: the front door every request crosses. Then the frontend's asset serving: the path that decides whether users see a page at all. The gateway will turn out to be a key element by position; the asset path, a key element carrying loads it never needed to carry.

The gateway: a key element by geometry

Almost every architecture I've touched has an API gateway, or its per-client cousin, the BFF. It sits between the client apps and every backend service. All traffic crosses it.

Run question 1 and it answers itself. Remove the gateway on paper and nothing reroutes. There's no alternate path around the front door; every route dies at once. Remember the transfer beam from the key-element section, the element with no understudy purely because of its geometry? That's your gateway. Not because it's badly built. Because of where it stands.

So question 3 applies, and the key-element playbook writes itself:

  • Keep it thin.Routing, auth, request shaping. No business logic. The less it changes, the less often you break it yourself.
  • Make the redundancy internal.N+1 instances behind dumb load balancing is designing the beam for a higher load, not pretending there's a second beam.
  • Give it the strictest SLO in the fleet, and the monitoring to match.

But question 2 is the sneaky one here, because the ties the gateway needs are inside it. A gateway is one shared pool of connections and workers sitting on top of a dozen upstreams. Now replay the opening cascade in miniature: one backend goes from 20 ms to 2 seconds, and its in-flight requests quietly eat the shared pool. Every other route starves. Recommendations takes down checkout again, only this time the collapse happens inside a single process.

The fix is the same discipline at a finer grain: every route gets its own tie and its own bulkhead. Per-upstream timeouts, per-upstream breakers, capped concurrency per route. A gateway with per-route bulkheads is a building of tied panels. Without them it's the stack of loose ones.

And the BFF split is a consequence-class move, whether or not anyone framed it that way. Carve the one gateway into per-client BFFs (mobile, web, partner API) and you haven't removed the key element; you've shrunk what each copy of it carries. Mobile's BFF dying is now a contained collapse with a named budget: one client platform, not all of them. Web checkout keeps selling. Same trick as cells, applied at the front door.

(In the pace-layers piece I looked at the gateway through a different lens: a shock absorber between fast-churning clients and slower services. Both checks apply. One asks how the gateway absorbs change; this one asks what happens the day it's gone.)

The frontend that serves its own assets

Default setup: one Next.js server (or any SSR origin) renders pages and serves the JS bundles, the CSS, the images. It's convenient: one deploy, one process, one URL.

Now remove it on paper. What still serves? Nothing — the user gets a blank tab. The origin was the only load path for everything: dynamic HTML, immutable bundles, a logo that hasn't changed in two years. That's the disproportion, and it got built by default. A deploy hiccup or a traffic spike on the rendering path takes down assets that have no business failing at all.

Sort the loads by consequence class, the way we sorted OrderService's dependencies, and the mismatch jumps out:

  • A content-hashed JS bundle is the most Class 1 thing in your whole system: immutable, cacheable, servable by any CDN edge, and returning visitors already hold a copy in their browser cache.
  • Dynamic HTML for checkout is origin work. Real key-element load.
  • Image optimization is worse than either: CPU-heavy resize work sharing a pool with your rendering. One product page full of uncached originals, and SSR starves. The gateway's shared-pool story again, one floor down.

The fix is the same discipline: give every load the cheapest path that can carry it. Bundles and images go to the CDN. Immutable assets are the easiest alternate path you will ever get; rerouting costs nothing and the failure mode shares nothing with your origin. Image resizing moves out of the rendering process (a separate service, or the CDN's optimizer) and takes its own bulkhead with it. What remains at the origin is the load only it can carry: dynamic render. Now it's a key element worth hardening — small, focused, holding a fraction of the traffic.

And notice the blast-radius budget you just bought. Origin down used to mean blank tab. With the shell and assets at the edge, origin down means stale or degraded dynamic content while the app still loads. That's a named budget: we serve, we degrade, we don't disappear.

✍️ Containment isn't a library

Here's the whole transfer in one table.

| Civil engineering | Distributed systems | The question it answers |
|---|---|---|
| Tie forces / catenary action | Timeout, bounded retry, backpressure, circuit breaker | When this connection is stressed, does it degrade gracefully or tear the caller apart? |
| Alternate load path (notional removal) | Failover / redundancy that genuinely re-routes; cells | If this disappears, what carries its load — and how big is the damage? |
| Named damage limit (15% of floor) | Explicit blast-radius budget (one cell, one region, X% of traffic) | What's the most failure we're choosing to tolerate? |
| Key element design | Isolated, hardened, highest-SLO critical dependency | What has no fallback, and so must not be allowed to fail? |
| Notional removal, run continuously | Chaos engineering (Chaos Monkey and descendants) | Which element is secretly a key element? |
| Consequence classes | Tiering dependencies by blast radius | How much of this discipline does |

But the table isn't the point. The reframing is. Civil engineering, after it buried four people under a corner of Ronan Point, decided containment could not be a component. It had to be a * design check*: run on every element, sized to consequence, mandatory by law.

We can run the same check. Better, even: continuously, against the live system, instead of once on paper. That's what chaos engineering is, once you see it through this lens.

Next time you add a dependency (or better, next time you sit down to understand a system you already own), go element by element and ask the three questions. Find the load path. Size the tie. Name the key element. It's the first pass I now make on any system I inherit.

Most dependencies will need only the cheapest answer. Making everything unkillable was never the goal. The goal is to know, before the incident instead of during it, which question each dependency has failed.

Containment isn't a library you install. It's a design check: if this element disappears, what carries its load?