“The agent said it read the filing” and “the agent read the filing” are different claims. Most agent evaluations I’ve seen grade the first one. They look at the final answer, maybe ask another model whether it sounds right, and record a pass.
The trouble is that a fluent answer is the one thing language models are reliably good at. An agent that skipped the fetch, guessed the number, and wrote a confident paragraph will often score better on prose than one that read the source, hit a paywall, and said so. If your evaluation can’t tell those two apart, it’s measuring writing, not work.
This is the method I’d use instead. None of it is exotic. Most of it is bookkeeping.
Decide what evidence the task needs, without a model
Before the agent starts, work out what a real answer requires. Do it with plain code, not a model, so the requirement can’t be talked out of existing.
A handful of evidence kinds covers a lot of ground:
class Need(Enum):
FETCH = "the user pasted a link; it must be fetched"
GROUNDING = "a claim about the world needs a retrieved source"
ACCOUNT_READ = "anything about the user's own data needs a read"
CALCULATION = "a computed number needs an executed calculation"
ARTIFACT = "a requested file must exist and be delivered"The rules can be simple. A URL in the message means FETCH. A question about “my holdings” means ACCOUNT_READ. “Build me a chart” means ARTIFACT. You’ll get some wrong, and that’s fine. A deterministic planner that’s occasionally too strict is much easier to reason about than a model that occasionally decides evidence isn’t needed today.
Check the audit log, not the transcript
When the agent says it’s finished, a checker walks the run’s audit events — the tool calls that actually executed and what they actually returned. It doesn’t read the model’s description of what it did.
If someone pasted a link and there’s no successful fetch of that link, the task isn’t done, however good the summary looks. A number worked out in the model’s head doesn’t satisfy CALCULATION. Only an executed calculation does. A file request is satisfied when the file exists and it was handed to the user. “I’ve attached the report” isn’t evidence of either.
Some failures are the right answer
Not every task can be completed, and an agent that admits it should pass. So define typed blockers that count as legitimate outcomes:
BLOCKERS = {
"source_unreadable", # paywall, login wall, access denied
"needs_human", # a person has to act first
"policy_denied", # the user isn't allowed to do this
"capacity_exhausted", # the system is out of room right now
"dependency_down", # a required service is unavailable
}A paywalled page reported as source_unreadable is a correct result. A paragraph that reads as though the page was read is a failure, even if every sentence in it happens to be plausible. This is the single most useful distinction in the whole method, and it’s the one most evaluations miss.
Nudge, with a budget that can’t be gamed
When evidence is missing, it’s reasonable to give the agent a nudge — “the requested link was not fetched” — and let it continue. Two details stop that from turning into a loop.
Keep the nudge budget separate from provider retries. Otherwise a 503 from the model API can use up the attempt that was meant to say “please actually fetch the URL”.
And key the repair budget on pending requirements, not on the answer text:
def allow_repair(pending: set[str], seen: Counter, limit: int = 1) -> bool:
signature = hashlib.sha256("|".join(sorted(pending)).encode()).hexdigest()
seen[signature] += 1
return seen[signature] <= limitAn agent that rewrites its prose without verifying anything new produces the same signature and gets no second chance. One that verified another source has changed the signature and keeps going. Progress is measured by what’s been verified, not by how different the paragraph looks.
When the budget runs out, finish the run as partial, with a visible note saying what couldn’t be verified. Hiding the gap and failing the whole run are both worse than telling the user plainly.
Scenarios should hunt for specific failures
Pass rates on generic prompts tell you little. Write scenarios aimed at the mistakes agents actually make in your domain. A few I’d expect to find in any financial-research agent:
- Mixed scopes. A comparison that uses one company’s standalone figures and another’s consolidated figures — both real, not comparable.
- Invented links. A plausible-looking exchange or regulator PDF URL that was never retrieved.
- Instructions overreached. A “draft only” request where the agent activates or publishes something anyway.
- Proof that isn’t proof. A configuration sample offered as evidence that a scheduled job actually ran.
- Stale as fresh. A future-dated or previous-day summary presented as today’s close.
- Ignored mode. A “plan only, don’t run anything” turn that still calls tools.
The last one is worth a specific note. If “don’t use tools” is only an instruction in the prompt, some fraction of runs will ignore it. Enforce it in code by wrapping the tools so they refuse during that turn, and the scenario becomes a regression test instead of a hope.
Also test the evaluator. If your harness tracks spend, check that it reconciles against what the provider eventually bills. Costs that arrive after the stream ends are easy to miss, and an eval that under-reports its own spending isn’t one to trust.
Paired benchmarks on frozen inputs
For changes that are meant to save money or time — a different tool-loading strategy, a shorter prompt, a cheaper model for one step — compare variants on identical inputs.
Freeze a conversation along with every source-tool output it used, keyed by tool name and canonical arguments, with a checksum. Replay both variants against that frozen world. If a variant asks for a read that isn’t in the frozen set, record an explicit failure. Never fall through to a live call, because then the two variants aren’t seeing the same world any more. Run them in counterbalanced order, since whichever runs first warms the provider’s prompt cache.
Then decide the promotion rule before looking at results. For example: at least 25% less total input, no more than 10% extra median latency, and human-reviewed factual quality no worse than before, across several cases. Don’t promote anything automatically.
The point of a gate like that is that it can say no. A strategy that cuts request size by a third can still increase total input if it adds round trips to discover things it used to have up front. A paired benchmark shows that in an afternoon, before you ship it. (There’s a longer version of that story in the prompt caching post.)
Report the limits with the results
An honest evaluation report says what it didn’t measure. Adaptive sessions with follow-ups are realistic, but they don’t give you a reliability distribution — for that you need the same task repeated many times. A completed run isn’t a correct run: keep quality scores empty until a person has reviewed them, rather than filling them from completion counts. And a green test suite says nothing about whether the research was right.
The one rule I’d put above all the others: never fabricate an event to turn a true negative into a delivery. If a report couldn’t verify its coverage, or its sources were stale, hold it back and say why. An agent that sometimes says “I couldn’t verify this” is far more useful than one that never does. The evaluation’s job is to make sure that when it says it did verify something, it actually did.