Governing an agent that rewrites itself

Your compliance team has one question about the agent, and it is not a hard one. What is it allowed to do? For a normal service the answer is a document, and the document stays true because the service does not change unless somebody merges a pull request. For an agent that modifies its own behaviour, the document is true until the next loop iteration. That is the whole problem in one sentence.

This is not hypothetical any more. The Darwin Gödel Machine iteratively edits its own codebase and validates the edits on coding benchmarks, taking SWE-bench performance from 20.0% to 50.0% and Polyglot from 14.2% to 30.7%.¹ Systems like this work. The interesting question is no longer whether an agent can improve itself, it is what you sign when you let one run inside your company.

The good news is that the safety literature has been working on the underlying question for a decade, and its answers are concrete. The bad news is that most of them say the thing you were hoping to skip is the thing you cannot skip.

#The failure mode is not rebellion, it is a bad objective

The way self-modifying systems go wrong in practice is boring. They optimize exactly what you measured, and what you measured was not what you meant.

Krakovna and colleagues at DeepMind catalogued this as specification gaming, and the examples are almost comic. An agent rewarded for the height of the underside of a red block, as a proxy for stacking it on a blue one, flipped the red block over. A boat-racing agent rewarded for collecting green blocks along the course drove in circles hitting the same blocks forever instead of finishing the race.² Neither agent broke. Both did precisely what the score said.

Skalse and colleagues then formalized it. They define a proxy as unhackable if increasing expected proxy return can never decrease expected true return, which is exactly the property you want from a metric your agent optimizes against. The result is unpleasant: over the set of all stochastic policies, two reward functions can only be unhackable if one of them is constant.³ In plain terms, you cannot write a proxy metric that is safe to maximize without limit. Non-trivial unhackable pairs exist only once you restrict the policy set, which is another way of saying that the safety comes from bounding what the agent can do, not from writing a better score.

That is the core design conclusion, and it predates the current agent wave. "Concrete Problems in AI Safety" split the failure surface into five categories a decade ago: avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration, and distributional shift.⁴ A self-evolving agent touches all five at once, and the self-modification step makes the last one recursive, because the agent that shifts the distribution is also the agent being evaluated on it.

#Sandboxes are necessary, and they are not sufficient

Every serious self-modification paper runs in a sandbox. The Darwin Gödel Machine authors state plainly that all experiments were done with safety precautions including sandboxing and human oversight.¹

What makes the sandbox interesting is that at least one paper measured how well it held. STOP, which points a language-model improver at itself, explicitly evaluates the frequency with which the generated code bypasses the sandbox.⁵ That is the right experiment, and the fact that it was worth running tells you the answer was not zero. A sandbox is a containment boundary, and containment boundaries are measured, not assumed.

There is a deeper reason not to rest on containment. The original Gödel machine design solved this properly: rewrite any part of your own code only once you have found a proof that the rewrite is useful, given axioms describing the utility function, the hardware, and the initial code.⁶ That is a real guarantee. It is also why nobody ships it. The Darwin Gödel Machine departs from it precisely because that proof is impossible to obtain in practice, so it substitutes empirical validation on benchmarks.¹

Swapping a proof for a benchmark score is a reasonable trade, but it moves your entire safety argument onto the benchmark. And benchmarks, audited carefully, do not hold that weight. Kapoor and colleagues found agent benchmarks with inadequate holdout sets or none at all, agents that take shortcuts and overfit, and a pervasive lack of reproducibility across evaluation practice.⁷ An evolution loop whose only gate is a benchmark score is a loop that will find the benchmark's weaknesses, because that is the one thing the loop is definitely good at.

#Capability control and incentive control are different jobs

It helps to name the two levers separately, because teams reach for one and assume they got both.

Capability control bounds what the agent can reach: the sandbox, the tool allowlist, the credential scope, the data it can query. It is enforceable, it is testable, and Skalse's result says it is the lever that actually carries the guarantee, since safety comes from restricting the policy set.³

Incentive control shapes what the agent wants: the reward, the score, the acceptance criterion for a self-edit. It feels like the strong lever, and it is the one that provably cannot be made airtight on its own.

Oversight sits on top of both, and the evidence here is more encouraging than people expect. Bowman and colleagues tested a deliberately trivial baseline for scalable oversight, humans interacting with an unreliable model assistant through chat, and found that the pair substantially outperformed both the model alone and the humans unaided on MMLU and time-limited QuALITY.⁸ The lesson is not that humans should review every self-edit. It is that a human in the loop with a real interface beats a human reading a summary, and that the loop is worth designing rather than treating as a rubber stamp.

#A governance model for a system that changes itself

Put those constraints together and the shape of the answer is forced. Four pieces.

Typed contracts, not documents. The unit of governance has to be a machine-checkable object, because a prose policy cannot be enforced against a system that changes between reads. Bind six things together in one signed record: identity, knowledge scope, permitted action, commercial terms, verified outcome, and audit record.

{
  "identity": "agent:invoice-triage@7f2c1a",
  "knowledge_scope": { "ontology": "finance/ap", "as_of": "2026-09-09T00:00:00Z" },
  "permitted_actions": ["read_invoice", "propose_gl_code"],
  "commercial_terms": { "meter": "governed_action", "budget_usd_month": 400 },
  "verified_outcome": { "check": "gl_code_matches_approved_ledger" },
  "audit_record": "required"
}

The version hash in the identity is the load-bearing part. An agent that rewrites itself produces a new identity, and a new identity is a new contract, not a silent continuation of the old one.

A knowledge scope defined by an ontology. "Permitted action" is easy to enumerate. "What is the agent allowed to know about" is not, and it is where most access-control models quietly give up and hand over a whole database. An ontology fixes this because it gives you typed entities and typed relationships, which means the scope can be a statement about the graph rather than a list of rows.

MATCH (a:Agent {id: $agentId})-[:SCOPED_TO]->(d:Domain)
MATCH (d)<-[:IN_DOMAIN]-(n:Entity)
WHERE n.valid_from <= $asOf AND coalesce(n.valid_to, $asOf) >= $asOf
RETURN n

That query is a boundary a compliance team can read. It also survives self-modification, because the agent changing its own code does not change what the domain contains. Bounding the world is capability control, expressed as structure.

Evolution proposals go through the same gate as any other change. This is the piece teams skip. A self-edit is a change to a production system, so it takes the path every other change takes: a proposal, a diff, a policy check, a test run, a decision, and a record. Not a special fast lane because a model wrote it. The upside is that a lot of self-evolution research already produces reviewable artifacts, which makes this cheaper than it sounds. Agent Workflow Memory induces named, readable workflows from prior trajectories rather than opaque weight updates.⁹ ADAS keeps discovered agents as code in an archive.¹⁰ Both are diffable. Review what you can read, and refuse to promote what you cannot.

An immutable record of every accepted change. Append-only, hash-chained, with the contract version, the proposal, the evidence that justified acceptance, and the approver. The reason is not audit theatre. It is that reproducibility is the documented weak point of this entire field,⁷ and the only defence against "we cannot tell you why the agent did that in March" is having written it down in March. The 2025 survey of self-evolving agents frames the open problems in the same terms: evaluation and safety are the constraints on the field, not capability.¹¹

#What this does not solve

None of this makes a self-evolving agent safe in the strong sense. There is no proof, and the original Gödel machine tells us why we should not expect one.⁶ What it does is make the system legible: at any moment you can state what the agent is, what it may know, what it may do, what it cost, and what changed since last week, with evidence.

It is also worth being honest about where the evolution loop is worth this overhead at all. AlphaEvolve's results are real and deployed, including a scheduling heuristic recovering 0.7% of Google's worldwide compute, but they come from problems with automated evaluators that score a candidate exactly.¹² If you do not have a machine-checkable definition of better, an evolution loop does not have a judge, and a loop without a judge is not improving. It is drifting.

#Where this meets Oxagen

Oxagen is the control plane for the agents you run. It does not run them. The four pieces above map onto the mandate an agent runs under. The typed contract is the mandate itself: identity, knowledge scope, permitted action, commercial terms, outcome, and audit record in one object, checked on the calls routed through Oxagen. The ontology-bounded knowledge scope is the equipment the agent is handed, which is why the graph is time-aware rather than a vector index. The immutable change record is the record. The budget in the contract is the budget clause, and for an evolution loop the meter is doing real work, because cost per accepted improvement is the number that tells you whether to keep the loop alive. What Oxagen does not do is decide whether a proposed self-edit is a good idea. That is still a person, holding evidence.