Which Model, When, Where — A Strategy for Targeted Premium Deployment

retrospectivemulti-modelmodel-selectionrefactoringcost

The session limit notification hit in Sprint 243.

Gatekeeper was mid-run on SHA-pinning eight third-party Actions across sixteen call sites. What was left in the .out file was two lines: "session limit · resets 7:10pm." The agent had stopped mid-work. The pin diff was still sitting in the working tree. My reaction was less surprise than "Oh, right. This model."

Fable 5 launched on June 10, 2026. I switched that same day. And the very first sprint directive I gave was: "Analyze every security issue and improvement point in the codebase, then build a sprint plan."


Problem

A more capable model was available. But whether it was always the right choice was a different question.

On Launch Day, I Had It Audit the Entire Codebase

Using Fable 5's first assignment as a full codebase audit was, in retrospect, the right kind of test. It wasn't "let's see if it's better" — it was closer to "let's give it the hardest thing possible."

The result was ADR-030. The tenth permanent ADR for AlgoSu.

A three-axis parallel scan — security surface, code quality, CI and infrastructure — ran first, and every suspicious conclusion was verified by directly reading the code. Five false positives were filtered out in that process. The Saga compensation transaction was flagged as absent, but it was there: compensateGitHubFailed/compensateAiFailed. Monaco lazy-load was flagged as unapplied, but next/dynamic had already handled it. Separating exploration (coverage) from verification (accuracy) is what let the false positives get caught.

The final verdict was zero high-risk findings. It was also confirmation that the baseline posture was solid. Three medium, five low, seven improvements. And from that, a five-sprint roadmap — S239 through S243 — was set.


Five Sprints Ran

  1. S239@Public() SSOT · Events API DTO validation · Prompt trust-boundary isolation · HeaderSanitizer
    완료

    Security Quick Wins

  2. S240Encryption key rotation runbook · DLQ redrive runbook (docs only)
    완료

    Operations Runbooks

  3. S241study.service 823 lines → 5 services · saga-orchestrator 516 lines → 3 modules
    완료

    BE Decomposition

  4. S242AddProblemModal 805 lines → 4 files · settings 844→252 · Redis logging alignment
    완료

    FE Decomposition

  5. S243Action SHA pinning for 8 actions · CI inline script extraction · CSP spike (deferred)
    완료

    Supply Chain · CSP · CI

The roadmap was consumed as planned. But what stood out more than performance was cost.

Token consumption was noticeably different. Deeper reasoning means the model holds a wider context in a single pass — and that means higher token consumption. Hitting the session limit in Sprint 243 — with Gatekeeper stopped mid-chain on the SHA-pinning work — was a first. The first session interruption since switching to Fable 5.


Decision

Keep daily sprints on Opus/Sonnet. For large-scale refactoring cycles, inject the premium model as a targeted deployment.

Regression Lives at Extraction Boundaries

Running Sprints 241 and 242 surfaced a repeating pattern.

Regression in decomposition refactoring concentrates at extraction boundaries.

When study.service.ts was split into five files, what was wrong wasn't the logic inside any individual file. It was the boundaries between files. The redis.keys O(N) issue had existed in the code before the split — the split was what brought it into Critic's view. The same held in the FE decomposition. Translation key wrapping was caught by auto-critic; setError(null) initialization dropping across component boundaries was caught by the full-base merge gate at R1; derived state handling was caught at R2. All five rounds of per-commit review had missed every one of them. R3 was the clean pass.

A more capable model has the potential to catch this in a single pass. If a finding that surfaces at auto-critic or merge gate R1/R2 can be resolved at the initial commit review, those round savings translate directly to cost savings. Taking the ADR-030 five-sprint roadmap as an example:

Critic Cumulative Rounds (S241~242)
15 rounds
BE 7 + FE 8 (including merge gates)
Boundary Regression Findings
6 findings
Auto-critic + merge gate R1/R2 — all past per-commit review

Every boundary regression was caught by auto-critic or the full-base merge gate — none by per-commit review. If a premium model's deeper reasoning catches them at the initial review stage, that eliminates the cost of cycling through multiple auto-critic rounds and merge gate iterations.

I'll acknowledge this is a hypothesis, not a measured outcome. But what I felt running through the ADR-030 roadmap was that a model already aware of the pattern "regression concentrates at boundaries" tends to scrutinize boundaries more carefully from the start.


The Conditions Under Which Premium Models Have ROI

The core of the strategy is selective deployment.

Daily sprints — feature additions, bug fixes, documentation updates — are well-served by Opus/Sonnet. Adding code on top of established patterns is where the difference in reasoning depth is least visible. Cost efficiency dominates.

Premium models earn their ROI in different circumstances:

  1. "Work where normal models cycle through multiple merge-gate rounds" — Large decompositions with many extraction boundaries, security audits, cross-cutting refactoring. Round savings map directly to cost savings.
  2. "Sprints that repay technical debt principal" — Like ADR-030: a concentrated effort to address accumulated issues. One-time cost here raises the baseline quality of subsequent normal-model sprints.
  3. "Starting from scratch" — Full codebase audits, new domain design. Work that begins with no context is where reasoning depth differential shows most clearly.

That's the core insight from these six sprints. Investing in a premium model isn't just for that sprint — it's to raise the baseline for the next n sprints. Like paying down debt: reducing the interest (round costs) by retiring the principal.


Caveats — Where This Strategy Breaks Down

Interpreting this as "always use the premium model" is wrong.

There are a few preconditions.

First, wholesale refactoring has rapidly diminishing returns. The ADR-030 decompositions were effective because there was clear "principal" to pay — 823 lines, 516 lines. Further decomposition of an already well-structured codebase carries regression risk with minimal gain. Deploy where the pain is explicit, not everywhere.

Second, bigger doesn't unconditionally mean more accurate. The ADR-030 decomposition was ultimately executed by normal-model agents (the Opus/Sonnet-based agent chain). Fable 5 designed and audited; regular agents handled execution. Separating review from execution is itself a strategy.

Third, cost is real. Hitting the session limit in Sprint 243 was a warning. Running every sprint on a premium model raises token consumption, session interruption risk, and cost together. Selective deployment is why the strategy has to remain selective.


Result

What six sprints taught me wasn't a model's limits — it was the importance of timing the deployment.

The Previous Post and This One

In Toward a Model-Agnostic Harness, I wrote about building a system not bound to any particular model — bringing in Critic as an addition rather than a replacement, and moving toward a harness where any model can occupy any seat.

This post is its sequel. The question after achieving that freedom — "so which model, when, where?" is what this one answers.

That freedom has to come first. While you're locked to one model, there's no choice to make. Only after the harness supports model swapping, only after any model can occupy the Critic seat — does strategy become possible. Timing becomes a thing you can choose.

The day Fable 5 launched, I deployed it immediately against the full codebase audit. The results were good. But more important than the results was what came after: knowing when to reach for it again. And that judgment settling into a frame not of performance comparison, but of technical debt repayment timing.

When the next large-scale refactoring becomes necessary, I'll reach for the premium model again then. Not now. The boundaries are already clean, and daily sprints are running well on top of them.

Related ADRs

Decision records behind this post

sprint-238

Codebase-wide Security & Improvement Audit + Sprint Roadmap (ADR-030)

First sprint after switching to the fable model. Started via /start argument 'analyze all code for security issues and improvements → list them → establish a sprint plan'. After a 3-axis parallel exploration (security surface / code quality / CI·infra), every key suspect item was re-verified by reading files directly per the sprint-99 lesson — 5 preliminary findings turned out to be misjudgments/already-resolved/downgraded (Saga compensation exists, Monaco lazy load already applied, SP196 GIN already resolved, ShareLinkGuard sound, key naming is a convention), and 1 new finding was added (POST /api/events is unauthenticated + plain interface, so ValidationPipe performs zero validation). Conclusion: zero High Risk; 3 Medium (S-1 missing @Public, S-2 events DTO, S-3 CSP unsafe-inline) + 5 Low + 7 improvements codified as ADR-030 (new permanent ADR, KR+EN) with a remediation roadmap for Sprints 239–243 (quick wins → runbooks → BE decomposition → FE decomposition → supply chain). Verification: 6 ADR gates pass (index permanent 10, EN 186/186, links 0, i18n 0, doc-refs clean, conversion PASS), Critic (Codex gpt-5.5 --base aea9528) R1 CLEAN, PR #426 CI green auto-merge squash 22da1f3.

sprint-241

BE decomposition — ADR-030 Q-1 (BE) + Q-2

Sprint 3 of the ADR-030 remediation roadmap. Two behavior-preserving refactors completed — coverage thresholds held + Critic auto 2 rounds → 4 findings fully fixed. Q-1 (BE): gateway study.service.ts (823 lines / 28 methods) → 5 domain services (StudyMemberService, StudyStatsService, StudyAccessService, MembershipCacheService, residual study.service) + shared types. StudyController injects 3 services directly (zero delegation boilerplate). All 5 new services + controller at 100% coverage. Q-2: saga-orchestrator.service.ts (516 lines) → SagaQuotaService (AI quota + CB ownership) / SagaTimeoutService (saga resumption + timer + lifecycle) / residual Orchestrator (advance*/compensate*). Zero call-site changes across 6 consumers. Critic auto R1 (gatekeeper output, base 241af57): M-1 redis.keys O(N), L-1 structured-logging pattern mismatch. Critic R1 (conductor output, base 606edb1): app.module.init.spec comment drift, fake-timer isolation gap. All 4 fixed; R2 (base 3dc4b13) CLEAN. gateway coverage 98.66/96.94/96.83/98.93 (threshold 98/95/96/98), submission coverage 98.69/94.13/98.93/99.04 (threshold 97/92/96/97) — both held.

sprint-242

FE decomposition + tests (ADR-030 Q-1 FE + Q-7 + Redis logging carry-over)

Sprint 4 of the ADR-030 remediation roadmap. Three oversized frontend files decomposed + Sprint 241 carry-over Redis logging aligned. D (Redis carry-over): 3 carry-over files + 3 adjacent files found by full grep — 6 files total patched to the 2-argument structured-logging pattern. Q-1 (FE): AddProblemModal (805 lines) → add-problem/ 4 files (206/271/263/220) + 20-line re-export shim + 65 tests; settings (844 lines) → 243 lines + 6 section components + 28 tests; edit (748 lines) → 465 lines + 5 section components + 22 tests (3 existing hooks reused). Critic auto-critic 5 rounds (D Low1 · A CLEAN · B/C M-1+L-1 · L-1-fix Medium 2 remaining · closure Low1) + merge-gate full-base R1 (M-1) → R2 (P3) → R3 CLEAN — all findings fully fixed. frontend 171 suites / 1769 tests, coverage 88.06/80.68/85.34/88.58 (threshold 81/71/82/83 held). gateway 854+ / 98.66/96.94/96.83/98.93; submission 387 / 98.69/94.13/98.93/99.04. tsc 0 / lint 0 across all 3 services.

Related Posts