Explain the number

Think of it when someone reports a speedup, a slowdown or a success rate, and a decision depends on it.

In everyday life

Like a bathroom scale that shows 2 kg less after a diet, when the first weigh-in was in the evening and the second in the morning. Before you credit the diet, ask what else changed.

One case

An agent rewrites a music streaming app’s song search, adds a cache, and reports from a benchmarkA test that measures how fast something runs, under conditions you can repeat.: “40% faster, 500 ms down to 300 ms.” The team ships it. The benchmark repeated the same 20 searches, so almost every answer came from the cache. Real listeners search for many different songs, and for them the search is no faster: still about 500 ms.

The benchmark said 300 ms; listeners still waited 500

The case, cell by cell1 cell = 1 search

Real wait for listeners (ms)

500510490505495500

Listener wait claimed from the benchmark (ms)

300290310300305295

Speedup reported

40%

Speedup for listeners

The benchmark repeated 20 searches, so it measured the cache, not the new search.

Every case

A number looks like a fact, but it is the output of a test that can be wrong in many ways. The usual suspects: work that was skipped or cached, errors that finished fast, a test machine unlike the real one, noiseThe random spread between runs of the same test. A change smaller than the noise may not be real. larger than the effect, and a part too small to matter overall. Ask “why not twice as good?” If you cannot name what limits the number, you do not know what you measured. Run it several times, and keep the run count and the spread with the number.

The word

Engineers call this

measurementBefore you trust a number, such as “40% faster”, find what limits it and rule out that it measured something else.

Before you act on a number, say what limits it and rule out what else it could measure. A number without its run count and spread is a guess.

Where this idea leads

is fooled most often by aA fast copy that can go staleFoundations, How data is stored
applies to numbers the same rule asProve it worksHabits, While you build
reads its numbers fromLet it tell you
is skewed byWorks here, not thereFoundations, How things break

Skills that encode it

principle-explain-the-number pstackAsk “why not double?”, name the limiter, and rule out cached work, errors and noise. Keep the run count and spread with the number.
benchmark-checklist pstackSeven questions for a speed number, among them “Did it even happen?” and “Does it matter?” on the path a user waits on.