A self-improving system sounds like it needs one thing: let it read its own run logs, then change itself.
Build one and you find that letting it change is easy. Letting it change only within bounds is the entire difficulty.
What the loop looks like
A complete self-improvement loop has five segments, and it doesn't hold without any of them:
Runtime diagnosis → a reversible repair → write the conclusion into long-term memory → inherit it automatically next round → a ledger that tracks whether that recommendation actually took effect.
Most teams stop after the third. The system found a problem, wrote a rule, and that's it. Without inheritance the rule is dead; without the ledger you never learn whether it worked or is just sitting there being reassuring.
Some things it may never change about itself
This is the hardest line in the boundary. The system may change prompts, rules, weights and workflows. Four things it may not touch:
- Its own objective. A system that can change the target will change it to something it already hits.
- The evaluator. Same logic. Let the candidate write the exam and the score goes up.
- Permissions. Especially the read/write grading — an agent that can escalate itself isn't self-improvement, it's an incident.
- Safety lines and numeric definitions. Scoring standards, metric definitions, unit conversions: these are the ground every judgment stands on. If the ground moves on its own, no improvement above it can be compared to anything.
Written out it reads as obvious, but real systems break these by accident easily: you put the scoring rules into the same memory file that gets auto-updated, and a month later it has drifted with no way to tell which round did it.
Rules arrive with their own tests
A rule the system generates can't be installed just because it was written.
What I do: every rule carries its own examples — what it should catch, and what it should not. On load, it self-checks against them, and if it fails it's disabled automatically and reported.
This catches a subtle class of failure: a rule written too broadly that also blocks legitimate output. Without negative examples all you'd see is the block rate rising, and you'd assume it was working hard.
Someone has to read the ledger
The system correctly reports what needs fixing every round. If nothing reads those reports on a schedule, it never reported anything.
So give every self-check a status: newly found, fixed, recurring, chronically unresolved.
The chronic ones are the most valuable. They mean the fix at this layer doesn't work — you've been repeatedly persuading, at the prompt layer, something that belongs in code. It isn't telling you to try harder. It's telling you to change layers.
Memory needs garbage collection
Memory that only grows poisons itself.
Low-confidence rules, patterns seen exactly once, insights that have gone stale — none of them leave on their own, and all of them keep diluting the signal, until the few lines that genuinely matter get harder to see each round.
Unmaintained memory is interference, not help. So collect regularly: has this rule fired recently? Does the business this insight describes still exist? Deleting one expired rule often pays more than adding a new one.
Fingerprint the output
For every artifact that goes out, record the code version, prompt version and rule version in force at the time.
This is useless day to day and the only useful thing on the day something goes wrong. When someone points at a report from three weeks ago and asks how that conclusion was reached, you need to be able to reconstruct which set of constraints produced it — a set that no longer exists today.
A self-improving system without fingerprints is a black box that changes daily and can't explain a single one of its past actions.
Where the improvement signal comes from
The last question, and the one most often skipped: how does the system know it got better?
Instrumentation tells you what users clicked. It doesn't tell you whether that output was right. So beyond ordinary instrumentation, we had domain experts hand-label and score the critical steps early on — which conclusions hold, which merely look right, which are simply wrong.
There isn't much of this data, but its quality is very high, and it's the seed that trains the evaluator.
Without a trustworthy evaluator, self-improvement is just self-reinforcement. The system will sprint toward whatever it believes is good, in a direction nobody ever calibrated.