Let it tell you

Think of it when code runs where you cannot watch it, such as on a server or on users’ phones.

In everyday life

Like a smoke alarm. You do not stand in every room watching for fire; the alarm watches and wakes you. Without one, the neighbours tell you, late.

One case

At 19:00 on a Friday, a food delivery app’s payment provider starts to refuse 1 in 3 card payments. The app shows a vague error, and no alert exists. The team finds out at 21:15, 135 minutes later, from an angry post on social media.

A third of payments failed for 135 minutes, and no alarm rang

The case, cell by cell1 cell = 15 minutes

Payments tried

290300310320330340340330320310300

Payments that went through

290200207213220227227220213207200

Minutes before anyone knew

135

Alerts sent

An alert on the failure rate would have rung at about 19:05.

Every case

Once code is live, it runs where nobody is watching. Every system fails somehow; the question is who notices first. Logs record what happened, so you can trace a problem afterwards. A metricA number a system counts over time, such as failed payments per minute. counts something over time, such as payments per minute. An alert watches a metric and calls a person when it crosses a line, such as “more than 5% of payments fail”. Without these, your customers are your alert system, and they report late, vaguely and in public.

The word

Engineers call this

logs, alertsMaking a running system report what it does: logsLines a program writes as it runs, one per event, with a time, such as “14:02 payment 881 refused: card expired”. of each event, and alerts that call a person when something goes wrong. You learn of a problem from the system, not from a customer.

Before you ship, decide how the system will tell you it is broken. Alert on what users feel, such as failed payments.

Where this idea leads

watches what happens after eachPut it liveFoundations, How teams ship
records eachSilence is not "no"Foundations, How things break
makes numbers that still needExplain the number
keeps the evidence that leads to theFix where it beganHabits, Before you ship

Skills that encode it

principle-fix-root-causes pstackIf the deciding fact took a deep dig to find, propose the cheap, always-on signal that would have shown it at first look.