One fall knocks the next

Think of it when one slow or broken part sits under many others, or an outage spreads far beyond where it began.

In everyday life

One stalled car on a bridge backs up the road, then the junction behind it, then the streets behind that, until drivers who never meant to cross the bridge are stuck too.

One case

At 12:00 a shop’s recommendation service slows from 0.1 to 10 seconds per answer. The product page waits for it with no time limit, so each page request holds a workerA program, or one part of a program, that takes one job at a time and does it. for 10 seconds. The server has 200 workers. By 12:03 all 200 are waiting, and checkout, which shares them, stops too.

A slow side service used up the workers checkout needed
The case, cell by cell1 cell = 1 minute
Free workers, a normal day
200200200200200
Free workers, this day
2001306000
Minutes to a full outage
3
Parts that were broken
1
Only the recommendation service was slow. Everything that shared its workers stopped.

Every case

Parts of a system share things: workers, connections, memory, a database. When one part slows down, the callers that wait on it hold those shared things longer, and new requests pile up behind them. Retries add more load. The failure spreads along every line of dependency, until parts with nothing wrong with them are down too.

The word

Engineers call this

cascading failureOne part fails, and its failure overloads or blocks the parts that depend on it, which then fail too. A small fault becomes a full outage.

Contain each failure where it starts: put time limits on calls, retry gently, and let a page work without its optional parts.

Where this idea leads