Deterministic Coding Agents: Every Turn on the Record

An agent touched fourteen files overnight. The diff is on the branch and the tests are green. Someone asks the obvious question in review: why did it change that file? Nobody can answer. The reasoning is gone, the tool calls are gone, and the only artifact left is the output.

Run the agent again on the same ticket and it produces a different patch. Now the review question has no answer even in principle, because there is no run to go back to.

This is a solvable problem, and it is not solved by making the model deterministic. It is solved by making the run replayable and the turn auditable. The research on what makes coding agents work in the first place points at the same artifacts that review needs, which is a convenient accident worth taking advantage of.

#The loop that works is an execution feedback loop

Every result that moved code repair forward has the same ingredient: an external, executable verdict fed back into the model.

Chen and colleagues showed this with Self-Debugging, where a model inspects execution results and explains its own code back to itself, a technique they compare to rubber duck debugging. It gained 2 to 3% on the Spider text-to-SQL benchmark, with a 9% gain on the hardest difficulty level, and up to 12% on TransCoder and MBPP when unit tests were available.¹ The interesting part is the sample efficiency: the loop matched baselines that generated over ten times as many candidates.¹

Xia and Zhang pushed the same idea into automated program repair with ChatRepair, which interleaves patch generation with immediate test feedback instead of the traditional generate-then-validate batch. It fixed 162 of 337 bugs at roughly $0.42 each, and learned from failed attempts as well as successful ones.² Reflexion generalised the pattern beyond code, storing verbal reflections on failed attempts in memory, and reported 91% pass@1 on HumanEval against 80% for the prior result.³

Read those three together and the mechanism is clear. The model is not getting smarter between attempts. It is getting information: an exit code, a stack trace, a diff that did not apply. That information exists as concrete artifacts at a specific point in time, and most agent implementations discard it the moment the loop moves on.

Those discarded artifacts are exactly what a reviewer would want.

#Sampling is where the determinism goes

The benchmark literature has been transparent about this for five years, in the metric itself.

Codex scored 28.8% pass@1 on HumanEval and 70.2% pass@100.⁴ The gap between those numbers is not a footnote, it is most of the measured capability. A large share of what a model can do is available only through repeated sampling. In a benchmark you keep the sample that passed. In your repository you keep the sample that ran, and you have no oracle telling you it was the good one.

Turning the temperature down does not close this. Ouyang and colleagues studied 829 code generation problems across three benchmarks and found that the proportion of tasks producing no identical test output across repeated requests was 75.76% on CodeContests, 51.00% on APPS, and 47.56% on HumanEval.⁵ They also found that setting temperature to zero reduces nondeterminism without eliminating it.⁵ Same prompt, same model, same repository, different patch.

So nondeterminism is not a defect to engineer away at the model layer. It is a property of the system, and the engineering response is to record which sample you got rather than to pretend you can predict it.

#Nondeterminism has a review cost, and it compounds

Three costs, each concrete.

You cannot bisect a decision you cannot reproduce. Standard debugging practice is to narrow a failure by running again with one thing changed. That technique requires the previous run to be a fixed point. If running the agent again produces a different patch, every step of the bisection is measuring the sampler instead of the change you made.

The grader is nondeterministic too. Luo and colleagues carried out the first extensive study of flaky tests, examining 201 commits that fixed flaky tests across 51 open-source projects, and catalogued the root causes: async waits, concurrency, and test order dependence dominate.⁶ Put a nondeterministic author in front of a nondeterministic grader and a green gate becomes a sample from a distribution, twice over.

And a passing gate is a weaker claim than it looks. Yu and colleagues found 345 patches in SWE-Bench that were marked as passing but were wrong, because the tests attached to the original pull request never covered the failure mode.⁷ "Tests passed" is a claim that needs its evidence attached, not a boolean to be trusted on sight.

#What an auditable turn looks like

The structure that survives contact with real review is four nested records, appended and never edited: session, turn, tool call, and gate result. Event sourcing is the right shape here for a boring reason. If every event is immutable and state is derived by replaying the log, then the log is the audit trail rather than a second artifact you have to keep in sync with one.

A single turn record, reduced to what actually gets asked in review:

{
  "session": "s_01J8QK4M2ZT",
  "turn": 7,
  "model": "claude-opus-5",
  "prompt_sha256": "9f2c1ab0…",
  "tool_calls": [
    { "seq": 1, "name": "read_file", "path": "src/billing/grants.ts", "result_sha256": "1a7b…" },
    { "seq": 2, "name": "apply_patch", "diff_sha256": "c40e…", "files": 2, "added": 31, "removed": 4 },
    { "seq": 3, "name": "run_tests", "cmd": "pnpm --filter billing test:unit -- grants.test.ts", "exit": 0 }
  ],
  "gate": { "name": "unit", "verdict": "pass", "log_sha256": "77d1…" }
}

Hashes rather than payloads keeps the record small and still verifiable, but only if everyone hashes the same logical record the same way. That is what RFC 8785, the JSON Canonicalization Scheme, standardises: a deterministic serialisation, property ordering, and encoding so that a JSON document has one canonical form suitable for signing and hashing.⁸ Without a canonicalisation rule, two services that agree on the content will disagree on the digest, and the audit trail quietly stops verifying.

Here is what each layer pins, and the review question it answers:

Note that none of these require the model to be deterministic. They require the harness to be honest.

#The parts you can actually pin

Some of the run is reproducible, and it is worth separating that from the part that is not.

The pinnable parts are the toolchain version, the container image, the repository commit, the prompt text, the model identifier, the sampling parameters, and the recorded input and output of every tool call. Record those and a colleague can replay the same sequence of actions against the same tree, even if a fresh generation would diverge.

The software supply chain has been down this road. Lamb and Zacchiroli describe reproducible builds as the practice of getting bit-for-bit identical results from the same source, and are direct about how hard it is in real projects: timestamps, build paths, and ordering all leak into outputs.⁹ The value is not aesthetic. It is that a third party can verify a binary corresponds to its source, rather than trusting whoever built it. Coding agents cannot reach bit-for-bit today, but the goal transfers exactly: someone other than the author must be able to check the claim.

There is also a design lever. SWE-agent's argument is that the agent-computer interface determines what an agent can do at all, and their purpose-built interface for navigation, editing, and testing produced a 12.5% pass@1 on SWE-bench where non-interactive approaches did far worse.¹⁰ A finite, designed action vocabulary is also a finite, loggable one. You cannot record an unbounded set of shell improvisations as cleanly as a fixed set of named tools.

Agentless makes the sharper version of the point. Xia and colleagues replaced the agent loop with three fixed phases, localise, repair, and validate, and resolved 32.00% of SWE-bench Lite at about $0.70 per issue, ahead of the open-source agents of the time.¹¹ A pipeline with fixed stages is trivially easier to replay than an open-ended loop. That is not an argument against agents. It is evidence that reproducibility and capability are not opposites, and that reaching for an autonomous loop should be a decision you can justify rather than a default.

#Where this meets Oxagen

Oxagen is the control plane for the agents you run. It does not run them, and it cannot make a model's sampling deterministic. What it does is keep the record complete. Every run is recorded frame by frame, each frame hash-chained to the one before it, and the seal signs the whole run, so anyone with the export can check a turn rather than take it on trust. The coding agent sits on the other side of that seam: it does the work and emits the frames, and Oxagen binds each step to the agent's mandate and the rule that answered its request. The honest claim is narrow and useful. You still get a different patch on a second run, but you can always say exactly what happened on the run you shipped.