Evaluating agents

A unit test checks that a function returns the right value. An agent test has to check something more awkward: that the model chose the right tools, in a sensible order, with correct arguments, within budget, and then said something true. The answer alone is not enough, because an agent can reach a correct sentence by a wrong route, such as escalating a ticket it should have answered from the help centre. It can also fail to reach it because a tool description changed, and nothing in a plain output check will tell you which.

The practical approach has two tiers. Replayed evals run in pytest with no network: recorded model responses are played back, and assertions check the trajectory. They are fast, free, deterministic, and they catch regressions in your code. Live evals call the real model against a small dataset of cases and score the results, with assertions where you can and a model-as-judge where you cannot. They are slow and cost money, so they run nightly or before a release, and they catch regressions in the prompt and the model.