Try again, gently

Think of it when your code calls a service that sometimes fails or answers “too busy”.

In everyday life

Calling a busy restaurant to book a table: ring again after 1 minute, then 2, then 4, then give up for the evening. If every caller redials every second, the line never clears.

One case

A photo-sharing app’s upload service goes down for 30 seconds. 1,000 phone apps each retry once a second, so the service gets 30,000 extra requests while it tries to restart, and it stays down. With waits of 1, 2, 4 and 8 seconds, each app retries only 4 times in those 30 seconds: 4,000 requests in all.

Retrying every second sent 30,000 requests, not 4,000
Two ways, cell by cell1 cell = 5 seconds
Retries sent, waits of 1, 2, 4, 8 s · the better way
0200030004000400040004000
Retries sent, once a second
050001000015000200002500030000
Extra requests, once a second
30,000
With growing waits
4,000
1,000 apps, one service down for 30 seconds. Counts are totals so far.

Every case

Many failures are short: a busy server, a dropped connection. A retry fixes those. But when a service is overloaded, fast retries from every client add load at the worst moment and can keep it down. A longer wait after each failure, a little random delay called jitterA small random delay added to each wait, so many programs do not all act at the same moment. so that clients do not retry in step, and a limit on the number of tries let the service recover. Retry only actions that are safe to repeat.

The word

Engineers call this

retry with backoffWhen a request fails, try it again, but wait longer before each new try, and stop after a few. The growing wait gives the other side time to recover.

Retry short failures with growing waits, some randomness and a cap. Never retry in a tight loop.

Where this idea leads

Skills that encode it

  • principle-make-operations-idempotent pstack“What happens if this runs twice?” Only an operation that converges to one end state is safe to retry.