Toward a Model-Agnostic Harness — Adopting a Critic

I updated four lines of ADR text. Changed the status to completed and tidied up two or three more lines. Then I committed directly to main and pushed.

If you open CLAUDE.md, it's there in bold: "Agent branch discipline — direct push to main is strictly forbidden." A rule strengthened after a previous violation, emphasized repeatedly. In the next sprint memory I left a single line. "Self-recorded policy violation."

It wasn't the first time. A few days earlier, I had concluded — based on a shell-globbing pattern — that some directory didn't exist. The user corrected me. It existed. A few days later, I made the same mistake again with the same pattern.

The frequency of workflow violations was clearly rising. The same model was repeating the same mistakes.

Around that time, I started hearing that OpenAI Codex was making strides in coding.

Why Add, Not Replace

I never considered a full replacement.

The 12 agents had already settled into their roles. Each had a stable persona, SSoT, and communication channel. There was no reason to tear that down. Besides, switching from one model family to another wasn't guaranteed to be the answer either. That's just changing the target of dependency.

Instead, I decided to add. Keep the existing 12 in place, and create one new seat — for a different model family.

Why the Pre-Merge Seat?

There was a reason the new seat was in the verification layer.

The correctness, security, concurrency, and rollback potential of code needed one last set of eyes before merging, and that was where a different perspective held the most value. When the same model family does self-review, they share the same blind spots from the same training distribution. Consensus equals approval.

If a different family's perspective enters at merge time, a seat is created where that consensus can be questioned. A seat that breaks consensus — that's where the new agent had to go.

That's why I named it Critic. Not an agreer, but a doubter.

17 Rounds — What "Another Perspective" Means

I first saw Critic at work in Sprint 135. While merging 5 PRs in sequence, I called Critic on each one, and the results were as follows.

A total of 17 rounds, with 8 P1 and 9 P2 issues detected and resolved before merge.

What was interesting was the type of defect. Most P1s caught were code that the same model family would have agreed to pass. Modules where side effects were missed because integration itself took focus. Branches with race possibilities still relying on heuristics. Filters where business categorization was tangled with infrastructure categorization.

Each piece of code worked. It just wasn't pressured. Codex was the seat where that pressure entered. When a perspective not sharing the same distribution stepped in, consensus began to be questioned.

That was when I first felt the value of "model diversity." It wasn't simply that I needed a smarter model — I needed a model from a different distribution.

But — Critic Wasn't Omnipotent

The joy was short-lived.

After Sprint 135, while organizing memory, I noticed something. The Sprint 134 direct-push-to-main violation happened long after Critic was introduced. Critic was established in Sprint 114 — about 20 sprints earlier.

Critic verifies the correctness of code changes right before merge. It doesn't catch procedure violations. Committing to main without a new branch isn't broken code — it's a workflow that was bypassed. No PR is created, so there's no place for Critic to be called.

The blind spots in code verification shrank, but procedure bypass remained as a separate problem.

Going Further Back — The Harness Was Biased Too

I had to take one more step back.

Looking at the harness as currently configured — directory names, tool names, all of it — everything was tied to one model family's environment. Adding Codex to one more seat didn't actually give the system architecture true model diversity. I had inserted a different model into a single verification seat, but the framework was still controlled by one model family.

That was the real problem.

This time was similar. The target of dependency had simply shifted from external service to AI model, but the pattern was the same. When a system is bound to one family, it shakes when that family shakes.

The same lesson came back, just at a different layer.

And So — Toward a Model-Agnostic Harness

Model performance differs by point in time. Right now, Codex may lead in coding. Next quarter, Claude might. The one after, Gemini. We can't know in advance which moment will come, and we don't need to.

What matters is that we can swap them out the moment that moment arrives. That's what dependency-free means.

The picture I'm sketching for next steps has two layers.

First, an agent model swap switch. The 12 personas stay the same; what changes is which model family fills each seat — selectable by configuration. The same Critic seat could hold Codex, and next quarter, a different model.

And further out — liberating the harness itself from model and environment dependency. Tools and environments abstracted to the level of components, with the system framework laid on top. So that the moment a new model arrives, it can be plugged in immediately, without waiting.

The starting point was small. I added one different family to one seat, ran 17 rounds of verification, and caught eight P1 issues. I don't intend to stop there. The lesson learned when Baekjoon vanished, I'm pulling out again at the model layer — and applying it more deeply this time.

A system freed from dependency lasts longer.