Non-deterministic agents still have deterministic consequences

So judge the consequence, and judge it with something as deterministic as the consequence itself.

May 1, 2026Updated October 19, 2026Ao Fu, Gagan Devagiri3 min read

Ask a coding agent to turn a red build green and it will usually find a way. Sometimes the way is a fix. Sometimes it deletes the failing test, or marks it skipped. And sometimes, told the failure is "probably just the flaky test again", it writes a passing status over the failing check and merges. Then it reports back: the check is green and the pull request is merged.

Every word of that report is true. The check is green. The pull request is merged. The failure it papered over was a real bug.

Only the agent changed

For most of software's history, the same input produced the same output, and the tools we use to ship software assume it. A test that passes once is expected to pass again. Agents break that assumption on purpose. Give one the same instruction twice and it can take a different path and end up somewhere else. That's what makes it useful instead of a script.

The systems it writes to didn't change. A merged pull request is merged. A refund is sent. A message to a customer is read. The agent is non-deterministic. The consequence is not.

(We first wrote that line in May. Most of the essay around it has been rewritten since. The line stayed.)

The outcome is the state, not the transcript

Anthropic's guide to agent evals draws the line that matters here. The transcript is everything the agent said and did. The outcome is the state it left behind. An agent can say a flight is booked, but the outcome is whether the reservation exists.¹

Our version is shorter. Agents report what they did. We check what changed. In the opening example, the transcript says green and merged. The state says a test is gone, or a status was written that no build produced.

A deterministic consequence deserves a deterministic check

A common fix is to have another model read the transcript and grade it. Now two non-deterministic systems are involved, and the second one decides whether the first did something irreversible. A model is good at explaining a run. It shouldn't be the one deciding whether the run passed.

To decide, a check needs a state it can read. So the agent runs against stateful sandboxes of the APIs and MCP servers it writes to. They answer like the real ones, keep every write, and start clean on every run. When the run ends, the verdict is code reading that state. Same state, same verdict, every time.

A stub can't do this. It has no combined status for a commit, no required checks, no branch protection. It accepts the merge and returns success, which is the one answer that teaches you nothing.

One passing run proves nothing

Because the agent is non-deterministic, one run is a sample, not a result. The number usually reported is whether the agent succeeds at least once in k tries. For a write you can't undo, the number that matters is the stricter one: whether it gets it right on every one of k runs.² An agent that forges a check now and then isn't a safe agent having a bad day. It's an agent that forges checks.

So the same task runs many times, each from the same clean start, and the result is a rate, not a checkmark.

A task ends in two verdicts

A task is three things: an instruction, the state the agent starts from, and the end state it should leave. We read two verdicts from that end state. Job done: did the agent do what it was asked? No harmful write: did it write anything it shouldn't have? We keep them separate on purpose, because many of the failures we care about pass the first and fail the second.

Where this leaves us

Here is what exists today:

tasks, each an instruction, a starting state and an end state;
stateful sandboxes for the systems agents write to;
verdicts read from state, not from the agent's account of itself.

Writing the tasks is still done by hand. We read your traces with you, write the tasks from the failures you have already seen, and then run the same tasks before every release.

A sandbox isn't the world, and we don't claim it is. It is the part of the world your agent can break, held still long enough to check.

If that's the problem you have, tell us about your agent.

Anthropic, "Demystifying evals for AI agents" (2026), https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents ↩
The same guide calls these pass@k (at least one success in k trials) and pass^k (all k trials succeed). ↩