A test for the agent itself

Think of it when you change a skill, a rules file or the model, and need to know whether the agent got better or worse.

In everyday life

It is like a baker who tries a new flour in all ten of the bakery’s usual recipes, not only the bread it was bought for. A better loaf means little if three of the cakes now fail.

One case

A team edits its review skill so the agent also flags missing database changes. They try it on one pull request, it works, and they ship it. A month later they run both versions on 10 old pull requests, each with a known bug. The old skill finds 9; the new one finds the bug it was made for, but finds only 6 in total.

The new skill caught its own bug and lost four others
The case, cell by cell1 cell = 1 pull request
Expected result
foundfoundfoundfoundfoundfoundfoundfoundfoundfound
Old skill
foundfoundfoundfoundfoundfoundfoundfoundfoundmissed
New skill
foundfoundmissedfoundmissedfoundmissedfoundmissedfound
Found by the new skill
6/10
Found by the old skill
9/10
Pull request #10 has the missing database change the edit was made for.

Every case

An agent is not fixed like code: the same promptThe text you give an AI model or agent as its instructions. can give different results, and a change that helps one task can hurt others. A test on the one case that started the change proves almost nothing. An eval runs many cases with the expected result known in advance, so you see the whole effect as one number. Models can act differently when a task looks like a test, so careful evals use tasks that read like real requests.

The word

Engineers call this

evalA fixed set of tasks with a known good result for each, used to measure how often an agent gets them right. You run it again after you change a promptThe text you give an AI model or agent as its instructions., a skill or the model.

Test changes to the agent the way you test changes to code: on a fixed set of cases, before and after. Trust the score across many cases, not one good run.

Where this idea leads

Skills that encode it

  • poteto-mode pstackIts Eval playbook tests a skill or prompt change on agent behavior, with organic-looking prompts, before promoting it.
  • principle-explain-the-number pstackFor an eval result: did every run do the task, and does the gap hold across trials and models?