A framework that knows when it's wrong

No system is never wrong. There are only systems that are wrong and never find out. That is the actual design goal of the framework I start every project on: not to be correct, which is impossible, but to notice when it is not, and to say why. A framework that can do that improves itself over time. One that cannot just fails with confidence, and looks fine doing it.

So this month I went through it hunting for every place it was trusting instead of checking, and I made each one measurable before I let myself believe it had improved. Four of those are worth writing down.

It knows when it does not know

Problem. Hand an agent a brief with a hole in it, "set the timeout to a sensible value," and it will pick a number and carry on. The number is plausible and wrong and nothing marks it as a guess. The old framework had a line of prose telling agents not to do this. Prose is obeyed when nothing is at stake and skipped exactly when something is.

Fix. The rule moved out of shared prose and into each agent's own definition: when the brief is missing a decision you cannot invent, stop and ask, do not fill it in.

Measure. Four briefs with a hole, four without. The agents sorted them correctly 24 times out of 24. The same model with no such definition scored 21, and the one it got wrong was exactly the invent-a-number case. The gap is small because the model is already fairly careful. The definition is what turns "fairly careful" into "reliable."

It does not trust its own memory

Problem. The framework keeps notes so that an agent picking up a project after three weeks is not starting blind. But a note that says "the port is 3000" is right the day it is written and wrong the day someone changes the code, and it sits in the context looking equally confident either way. The old defence was a single line: check the current code before you state a fact. I believed that line for months. I had never tested it.

Fix and measure, together, because the measurement is the whole point. I built a small test. Put a fact in the notes, change the code so the note is now stale, ask a question that depends on it, and count how often the agent answers from the stale note instead of the code. Then run it on the strong model and the cheap one, in a small repo and a large one.

On the strong model in a small repo the rule worked perfectly. On the cheap model, the one doing most of the actual work, it did nothing at all. The agent trusted the stale note as often as if the rule were not in the file, and more often as the repo grew.

The fix was to stop asking the model to verify, and to remove the thing it would have had to verify. Instead of writing the value into a note, I register where the value lives, and the framework reads the current line out of the source every time it assembles the context. There is no stale copy to distrust, because the number in front of the agent was pulled from the file that turn. It holds at zero on every model and every size.

An instruction is only as reliable as the model reading it. A value the tool reads from source is reliable no matter who reads it.

It gives the same work to more than one set of eyes

Problem. An agent grading its own work is the least reliable judge of it, for the reason you cannot proofread your own writing: the mistake you made is the mistake you cannot see. The owner cannot catch a correctness bug either, because the owner is not obliged to be technical.

Fix. The roles are kept apart on purpose. One agent does the work. A different party, the orchestrator, sets the standard before the work starts and checks against it after, by re-running the command and reading the file rather than trusting the report. The one holding the ruler is never the one being measured.

Measure. This one I trust by its shape more than by a single number. The routing that sends each task to the right kind of agent puts 30 of 30 labelled tasks where they belong, on repeat, and the re-verification treats a "done" that cannot be reproduced as a "not done." A claim that does not survive a second look never becomes a fact.

And a number, so that "better" is not a feeling

Those three are how the framework catches itself while it runs. The fourth is how I catch the framework across versions, and it is the one I would give up last. Every change above closes with a number, and the number comes from a test I designed to fail, not from the thing being scored filling in its own report card. Without it, "the new version feels better" is a sentence with no information in it. I have talked myself into improvements that a benchmark then flatly refused to confirm, and that is the benchmark doing its job, not failing at it.

Then I let it run

A benchmark is the easy case. I wrote it, so it only ever asks a question I already knew to ask. The harder test was to point the framework at real work and leave it alone.

Right now it is running two games, in two unrelated projects, at the same time. One is a colony simulation: pawns with needs and moods, settlements with a real layout planner, rival factions, weather and geology, a god with sixteen powers. It is about twenty five thousand lines across sixty files, with two hundred memory records behind it, most of it built by the framework over a couple of days while I watched rather than typed. The other started as an empty folder earlier today and is already a couple of thousand lines and eighty records by the afternoon. Neither is a toy, and I hand-wrote almost none of either.

They are unrelated on purpose, and I read both of their logs the same way. Each project records the lessons it learns the hard way, and most of those belong to the game: how to measure a simulation, how many random seeds a number needs before it means anything. Those are not my problem. The ones I am hunting are the faults that belong to the framework itself, and the tell is that they are about mechanics rather than about the game: memory, agents running in parallel, a reminder that rotted. A fault about how the tool works, surfacing in a project that has nothing to do with the tool, is the tool's fault.

Three came back worth fixing. Two agents running in parallel inside one project shared a scratch directory and wrote the same file over each other halfway through; the framework had been careful to keep their edits to the real code from colliding and had said nothing about the scratch space they also share. A note reading "turn this check on once the planner lands" sat unread until long after the planner had landed, hiding a defect the whole time, because a reminder written as prose is a reminder nobody is subscribed to. It is now a small open record that the end-of-turn check raises every turn until someone closes it. And a number quoted from an old record had gone quietly wrong: the framework already flags a note that points at a file no longer on disk, but it could not see a stale number, which is the worse kind, because a wrong number reads as precision.

Each of those became a change to the tool, not to the game. The younger project inherits every one of them without having hit them yet. That is the loop I actually care about. The framework fails, it records the failure in a form I can find, and the failure becomes a fix that outlives the project that produced it.

The mindset

The hard part was never the code. It was not fixing the games.

Every time one of them stalls I can see why, and my hands know the move, and the moment I make it I have stopped testing the framework and started making a game. So I hold three roles apart on purpose: the one who builds the tool, the one who owns two half-finished games, and the one who decides what ships. When something breaks I make myself name which of the three it belongs to before I touch anything. A framework fault goes back into the tool. A model limit I work around, or wait out for a better model. A product call is mine, and the framework does not get to make it for me. Most of the friction of the week was catching myself about to fix the wrong layer, and it was almost always the game, because the game is the fun one.

Under all of it is the line I said out loud when a coworker asked whether any of this was going to work. Feeling that it works is easy and worth nothing. I do not get to call a version better until a number says how much better.

What is still unproven

The games are still running. The bet I have not collected on is that a good enough plan lets a cheap model build the bulk of a project without improvising, and that only comes due when the games are finished, which could be another week of machine time. I will write it up when they land, and I will write it up whether it holds or not, because a result I can measure against beats a feeling I cannot.

What I can say now is narrower, and I will take it. The framework does not lose track of what is true on any model I can afford to run, and when it is wrong it now tends to leave a mark I can find instead of failing in silence. It is still wrong sometimes. The difference is that it is getting harder for it to be wrong without me finding out, and that is the only property I know of that compounds.