# Retirement Lab: evidence for agent-operated software development

Generated 2026-08-14 (America/Los_Angeles).

## Executive read

Retirement Lab is strong evidence for a narrow claim: a single operator writing no source code can build and maintain a substantial live product by specifying work and supervising agents. It is not evidence that agents produce defect-free code, that manual coding is obsolete in every setting, or that this workflow beats a strong conventional team on quality.

The history is best understood as three active development phases, separated by the repository's only two double-digit commit pauses:

1. **Bootstrap and core construction, February 23-April 3:** 545 first-parent commits in 40 calendar days.
2. **Productization and UX build-out, April 22-June 3:** 351 commits in 43 days.
3. **Platform expansion and hardening, June 18-August 14:** 176 commits in 58 days.

The most defensible results are:

- Throughput stepped down from 13.6 to 8.2 to 3.0 commits per calendar day, and from 15.1 to 9.2 to 5.2 per active day. The project did not sustain its greenfield pace.
- The mapped merged-run no-direct-intervention rate improved from 81.9% to 86.0% to 90.7%. This is the best repository-derived proxy for not returning to the keyboard after implementation started.
- Independent review or tests rejected 65.4%, 68.0%, and 80.0% of implementation-reaching runs at least once across the three phases. The agents were not simply generating code and merging it unchecked.
- Implementation used 460 Claude-family and 154 Codex-family runs. The measured blocking reviewer was a separate Codex gate: GitHub Actions early in Phase 1, then predominantly local Codex CLI review from March 17 onward.
- At each phase endpoint, product SLOC was 45,310, 119,342, and 158,063; test SLOC was 37,340, 83,513, and 124,199. Product complexity density fell from 115.4 to 96.5 to 90.4 `scc` branches per KLOC.
- Product-change test accompaniment rose from 80.4% in Phase 1 to 91.7% in Phase 2 and held at 91.3% in Phase 3. There is still no configured statement or branch coverage instrument.
- Change size became more concerning as throughput fell. The share of commits with at least 2,000 changed lines rose from 4.6% to 6.8% to 13.1%.
- A consistent semantic review of all 1,016 merged PRs found 262 underlying engineering defects addressed by 273 repair PRs. Excluding infrastructure and test-system failures leaves 183 product/customer defects, including 104 strict correctness bugs.
- Strict product repair work stayed essentially flat at 19.2%, 18.8%, and 19.4% of eligible product PRs. Origin attribution tells a different story: the 30-day strict defective-origin rate fell from 13.6% to 8.5% to 7.0%. Phase 3 has only 43 origins old enough for that comparison, so this is encouraging rather than conclusive.
- Change size is the clearest measured quality risk. Sixteen of 51 eligible origin PRs with 2,000+ changed lines later revealed a strict product defect (31.4%), versus 44 of 453 smaller PRs (9.7%).

The best article is therefore an instrumented case study about a different division of labor: the human chooses and specifies work; implementation, verification, review, and repair are mostly machine-operated. The bugs did not disappear. The keyboard work moved up a level.

## Scope, cutoff, and phase boundaries

- Git and PR cutoff: `origin/master` at `2cdd7383`, merged PR #1101 on 2026-08-14 at 07:27 Pacific.
- Git history: 1,072 first-parent commits from February 23 through the cutoff.
- Run telemetry: 711 deduplicated run records from the Mac and `thiago-linux`; 614 reached implementation, 599 passed, and 478 map to merged PRs.
- Phase run tables assign by local run creation date. One implementation-reaching run began on June 17 during the second commit pause and is outside the three phase totals, although its task PR #912 merged on June 18.
- Semantic review: all 1,016 merged PRs were classified from title, description, and changed files under one rubric. The 273 repair candidates were clustered into 262 underlying defects, or split when one PR fixed independent failures.
- Code-tree comparison: exact detached trees at PRs #520, #911, and #1101, measured with the same `cloc`, `scc`, and test-declaration rules.

| Phase | Active dates | Endpoint | Characterization |
|---|---|---:|---|
| 1. Bootstrap and core construction | Feb 23-Apr 3 | PR #520 | Core simulator, tax logic, initial product, testing, and agent workflow |
| Pause | Apr 4-Apr 21 | 18 inactive days | No first-parent commits |
| 2. Productization and UX build-out | Apr 22-Jun 3 | PR #911 | Design system, scenario UX, auth, observability, relocation, sharing, and mobile planning |
| Pause | Jun 4-Jun 17 | 14 inactive days | No first-parent commits |
| 3. Platform expansion and hardening | Jun 18-Aug 14 | PR #1101 | Public MCP, mobile platform, distribution, country/tax expansion, currency correctness, and product hardening |

These boundaries come from the work cadence, not a desired story. The 18-day and 14-day pauses are the only double-digit inactive gaps; the next-largest gap is five days. Mid-March QA adoption, the May 12 snapshot, and the July container migration are meaningful workflow events inside phases, but none is as strong a top-level development boundary.

The May snapshot represents the project from inception through PR #797, not the month of May. The defect ledger no longer depends on that old boundary: the pre-#798 PRs were reread under the same rubric as the later PRs. Treating May 12 as a development phase boundary would still mix a historical data-collection boundary with the shape of the work.

Thiago's zero-manual-coding statement is a self-report. Git cannot prove that a person never edited a file. The run ledger can independently show how often the orchestrated implementation path contains evidence that the operator had to return to the keyboard.

## Workflow being measured

The recurring production path is:

1. The operator selects a goal and creates or commits a spec.
2. An implementation agent works in an isolated worktree or container.
3. The verify phase runs tests and lint independently of the implementation conversation.
4. An independent code reviewer can request changes.
5. Findings are fed back to the implementation agent until the gates pass or the run exhausts its retry policy.
6. The system publishes and merges the PR.
7. Later production, QA, or analysis findings become new specs, creating the defect evidence in this report.

This is closer to operating a software-production system than to using autocomplete. The relevant unit is the complete run and its quality gates, not the number of lines an LLM emitted.

## Three-phase scorecard

### Delivery shape

| Metric | Phase 1 | Phase 2 | Phase 3 |
|---|---:|---:|---:|
| Calendar days in active phase | 40 | 43 | 58 |
| First-parent commits | 545 | 351 | 176 |
| Commits/calendar day | 13.62 | 8.16 | 3.03 |
| Active commit days | 36 | 38 | 34 |
| Commits/active day | 15.14 | 9.24 | 5.18 |
| Median commit churn | 175 | 467 | 395 |
| P90 commit churn | 1,338 | 1,603 | 2,183 |
| Commits with 2,000+ changed lines | 25 (4.6%) | 24 (6.8%) | 23 (13.1%) |
| Product lines added | 134,974 | 140,649 | 64,099 |
| Test lines added | 77,879 | 73,900 | 52,919 |
| Test additions/product additions | 0.577 | 0.525 | 0.826 |

Phase 1 contains 50.8% of all first-parent commits, Phase 2 contains 32.7%, and Phase 3 contains 16.4%. Lower later throughput is not itself a failure: the product was larger, work shifted toward wider platform changes, and inactive days became more common. But commits/day cannot support a claim that one operator equals a fixed number of engineers.

The change-size distribution is less comfortable. Median churn peaks in Phase 2, while the upper tail keeps growing through Phase 3. The later defect attribution strongly suggests that the 2,000+ line tail is a process risk, regardless of who or what wrote the code.

### Autonomy and machine QA

| Metric | Phase 1 | Phase 2 | Phase 3 |
|---|---:|---:|---:|
| Implementation-reaching runs | 254 | 269 | 90 |
| Passed runs | 248 (97.6%) | 264 (98.1%) | 86 (95.6%) |
| Runs mapped to merged PRs | 127 | 264 | 86 |
| Mapped runs with no direct intervention | 104 (81.9%) | 227 (86.0%) | 78 (90.7%) |
| Mapped runs with no broad intervention | 99 (78.0%) | 219 (83.0%) | 70 (81.4%) |
| Reviewer requested changes | 143 (56.3%) | 162 (60.2%) | 68 (75.6%) |
| Test gate failed at least once | 109 (42.9%) | 100 (37.2%) | 20 (22.2%) |
| Reviewer or tests caught something | 166 (65.4%) | 183 (68.0%) | 72 (80.0%) |
| Product commits accompanied by tests | 315/392 (80.4%) | 278/303 (91.7%) | 116/127 (91.3%) |

The direct autonomy result improves monotonically. The broad result does not: Phase 3 falls from 83.0% to 81.4% because the broad definition treats two-hour durable gaps as possible intervention, and July's container migration produced a temporary stall cluster. That is why the direct proxy is the primary keyboard measure and the broad proxy is a pessimistic sensitivity bound.

The QA mix also changes. Review catches rise sharply while test-gate catches fall. Their union reaches 80.0% in Phase 3, so the pipeline was more likely to reject an implementation at least once, not less. The repository cannot establish whether that reflects stronger review, harder full-spec work, fewer flaky test failures, model changes, or all of them.

### Code at each endpoint

| Same-method tree metric | PR #520 | PR #911 | PR #1101 |
|---|---:|---:|---:|
| Product SLOC | 45,310 | 119,342 | 158,063 |
| Support/tool SLOC | 6,021 | 25,600 | 29,872 |
| Test SLOC | 37,340 | 83,513 | 124,199 |
| Test/product SLOC ratio | 82.4% | 70.0% | 78.6% |
| Static test declarations | 1,818 | 3,774 | 5,770 |
| Product files (`cloc`) | 174 | 381 | 528 |
| Product complexity/KLOC (`scc`) | 115.4 | 96.5 | 90.4 |

The current tree has 187,935 non-test programming-language lines when product and support/tool categories are combined. Test volume did not grow in a straight line relative to product code, but the final ratio recovered to 78.6%, static test declarations more than tripled from the Phase 1 endpoint, and aggregate product complexity density fell 21.7%.

These are encouraging maintainability signals, not proof of maintainability. Generated volume can lower aggregate density, and `scc` complexity does not measure architectural coupling, readability, or correctness.

## What changed in each phase

### Phase 1: bootstrap and core construction

This was the greenfield burst: core simulation and tax behavior, inputs and results, shareable scenarios, an initial business surface, Playwright, linting, make targets, and dual AI review. Of 489 PRs, 274 were unmapped code changes, 87 were spec-authoring commits, 70 were full-spec implementations, and 57 were tasks. The relatively low 80.4% test accompaniment and 56.3% review-catch rate include the period when the workflow itself was still being assembled.

The phase produced 45,310 product SLOC and 37,340 test SLOC by PR #520. Its 13.6 commits/day is real repository activity, but it is primarily a bootstrap rate, not a sustainable mature-product baseline.

### Phase 2: productization and UX build-out

The second burst shifts from bootstrapping to deliberate product orchestration. The PR mix contains 170 full-spec implementations, 93 tasks, 41 spec-authoring commits, and only 45 unmapped code changes. Major work includes the design system and scenario redesign, authentication and observability, relocation analysis, glide paths, sharing, and mobile planning.

Product SLOC grew by 74,032 between endpoints, while product complexity density fell from 115.4 to 96.5 per KLOC. Product-change test accompaniment reached 91.7%. Throughput was lower than Phase 1, and it declined within the phase, but there is no repository pause or categorical workflow shift exactly at the May 12 snapshot.

### Phase 3: platform expansion and hardening

The third burst adds public MCP, currency contracts and sharing, mobile shell/IAP/push, Reddit distribution, country rankings and content, France-US and Vietnam tax work, FX correctness, Pro gates, and analytics. Of 176 PRs, 81 were full-spec implementations, 31 authored specs, 54 were unmapped changes, seven were tasks, and three were docs/content.

This is the slowest phase and the one with the fattest change-size tail. It is also the strongest phase on direct autonomy, machine-QA rejection, test additions relative to product additions, and aggregate complexity density. Both can be true: the pipeline looks more capable while its largest changes look riskier.

Within Phase 3, model labels begin changing on June 18 and execution begins moving to Linux containers at the end of June. July's stall spike and later recovery are operational subphases, not evidence for a fourth product-development phase.

## Intervention: when the operator had to return

"Intervention" means a keyboard action after implementation started. No single repository field records that fact, so the analysis uses two auditable proxies.

The direct proxy counts:

- an explicit operator response recorded during the run;
- a retry cap increased beyond the cap in force when the run began, with the run actually crossing the old cap; or
- a `failed` state, which is not retriable without a person restarting or replacing the run.

The broad proxy adds:

- an explicit operator-kill marker; or
- a gap of at least two hours between one durable phase event ending and the next beginning.

A long gap is not proof that a key was pressed, which is why it is a sensitivity bound rather than part of the primary metric. For a merged run, both scores include superseded ancestor runs. Restarting a failed implementation under a new run ID therefore cannot erase earlier evidence.

| Direct proxy, mapped merged runs | Phase 1 | Phase 2 | Phase 3 |
|---|---:|---:|---:|
| No intervention signal | 104/127 (81.9%) | 227/264 (86.0%) | 78/86 (90.7%) |
| Exactly one signal | 1/127 (0.8%) | 17/264 (6.4%) | 4/86 (4.7%) |
| Two or more signals | 22/127 (17.3%) | 20/264 (7.6%) | 4/86 (4.7%) |

| Broad proxy, mapped merged runs | Phase 1 | Phase 2 | Phase 3 |
|---|---:|---:|---:|
| No intervention signal | 99/127 (78.0%) | 219/264 (83.0%) | 70/86 (81.4%) |
| Exactly one signal | 6/127 (4.7%) | 14/264 (5.3%) | 9/86 (10.5%) |
| Two or more signals | 22/127 (17.3%) | 31/264 (11.7%) | 7/86 (8.1%) |

Across implementation-reaching runs, the intervention signals were:

| Signal | Phase 1 | Phase 2 | Phase 3 |
|---|---:|---:|---:|
| Explicit operator response | 1 | 30 | 4 |
| Retry-cap increase after crossing inferred initial cap | 32 | 13 | 4 |
| Failed-state recovery | 5 | 5 | 4 |
| Explicit manual-kill marker | 0 | 1 | 0 |

Signals overlap, so they do not sum to intervention-bearing runs. The explicit kill marker contributes only to the broad proxy.

Stall sensitivity was:

| Maximum observable durable gap | Phase 1 | Phase 2 | Phase 3 |
|---|---:|---:|---:|
| At least 1 hour | 37/254 (14.6%) | 40/269 (14.9%) | 15/90 (16.7%) |
| At least 2 hours | 29/254 (11.4%) | 33/269 (12.3%) | 13/90 (14.4%) |
| At least 4 hours | 23/254 (9.1%) | 26/269 (9.7%) | 12/90 (13.3%) |
| At least 8 hours | 21/254 (8.3%) | 21/269 (7.8%) | 8/90 (8.9%) |

July is the important outlier within Phase 3: 12/41 July runs had a two-hour gap and eight had an eight-hour gap. By August those figures were 1/38 and 0/38. The direct metric says autonomy improved; the broad metric correctly warns that the container transition temporarily made runs look less continuously unattended.

## Spec-managed work versus other PRs

The workflow provenance ledger distinguishes committed full-spec implementation, committed task implementation, spec authoring, and PRs that cannot be durably mapped to either committed-spec run mode. "Other code" does **not** mean manually written; it means the PR lacks a durable `spec implement` or `spec task` mapping.

| PR provenance | Phase 1 | Phase 2 | Phase 3 | All phases |
|---|---:|---:|---:|---:|
| `spec implement` | 70 | 170 | 81 | 321 (31.6%) |
| `spec task` | 57 | 93 | 7 | 157 (15.5%) |
| Other code PR | 274 | 45 | 54 | 373 (36.7%) |
| Spec/task authoring only | 87 | 41 | 31 | 159 (15.6%) |
| Docs/content | 1 | 2 | 3 | 6 (0.6%) |
| **All merged PRs** | **489** | **351** | **176** | **1,016** |

The workflow changed materially. Only 26.0% of Phase 1 PRs were mapped implementations of a committed spec or task, versus 74.9% in Phase 2 and 50.0% in Phase 3. Phase 1 therefore mixes product bootstrap with substantial pre-orchestrator or unmapped agent work. Phase 2 is the cleanest measure of the committed-spec pipeline. Phase 3 returns to more unmapped platform, content, and corrective work while almost eliminating short tasks.

| All-history PR shape | PRs | Median churn | Product PRs accompanied by tests |
|---|---:|---:|---:|
| `spec implement` | 321 | 878 | 315/317 (99.4%) |
| `spec task` | 157 | 242 | 125/155 (80.6%) |
| Other code PR | 373 | 174 | 260/307 (84.7%) |
| Spec/task authoring only | 159 | 164 | 0/4 |
| Docs/content | 6 | 242 | n/a |

Full specs are a distinct unit: 3.6x larger than tasks and 5.0x larger than other code PRs by median churn. They also have nearly universal test accompaniment. Intervention and reviewer-cycle comparisons are available only for mapped spec/task runs; assigning those run outcomes to the 373 other code PRs would fabricate telemetry that does not exist.

## `spec task` versus `spec implement`

`spec implement` consumes a committed `specs/*.md` file. `spec task` consumes a smaller committed `specs/tasks/*.md` instruction. They are different work units, not interchangeable prompt styles.

### Work size and testing

| Metric | P1 task | P1 spec | P2 task | P2 spec | P3 task | P3 spec |
|---|---:|---:|---:|---:|---:|---:|
| Implementation PRs | 57 | 70 | 93 | 170 | 7 | 81 |
| Median total PR churn | 195 | 1,097 | 299 | 851 | 279 | 958 |
| Median product churn | 75 | 449 | 111 | 452 | 55 | 479 |
| Product PRs accompanied by tests | 78.9% | 100.0% | 80.4% | 98.8% | 100.0% | 100.0% |

Full-spec implementations were 2.8x to 5.6x larger by median total churn. The task path stayed much smaller and was the persistent testing gap in Phases 1 and 2. By Phase 3 there are only seven task PRs, so its 100% accompaniment rate is not a stable trend.

### Run outcomes

| Metric | P1 task | P1 spec | P2 task | P2 spec | P3 task | P3 spec |
|---|---:|---:|---:|---:|---:|---:|
| Implementation-reaching runs | 74 | 179 | 93 | 176 | 6 | 84 |
| Pass rate | 100.0% | 96.6% | 100.0% | 97.2% | 100.0% | 95.2% |
| Reviewer or tests caught something | 78.4% | 59.8% | 55.9% | 74.4% | 16.7% | 84.5% |
| Mapped runs with no direct intervention | 89.5% | 75.7% | 96.8% | 80.1% | 100.0% | 90.0% |
| Mapped runs with no broad intervention | 86.0% | 71.4% | 93.5% | 77.2% | 83.3% | 81.2% |

The Phase 1 catch-rate reversal is a warning against treating mode as a causal variable: workflow maturity and task selection differed. In Phases 2 and 3, full specs are much more likely to be challenged by QA, which is consistent with their size and complexity. The six Phase 3 task runs are too few for a useful comparison.

## Independent reviewer and tests

| Implementation-reaching runs | Phase 1 | Phase 2 | Phase 3 |
|---|---:|---:|---:|
| Reviewer requested changes | 143/254 (56.3%) | 162/269 (60.2%) | 68/90 (75.6%) |
| Test/verify gate failed | 109/254 (42.9%) | 100/269 (37.2%) | 20/90 (22.2%) |
| Either mechanism rejected the run | 166/254 (65.4%) | 183/269 (68.0%) | 72/90 (80.0%) |

Examples show that the reviewer was finding consequential issues, not just formatting:

- The `auth-conversion-gates` review found that the anonymous run cap used a non-atomic check/increment and could be bypassed with concurrent requests.
- The `form-people-model` review found that migrated couple scenarios could still be saved without required partner fields and that Pro-only controls leaked outside the gate.
- The `shareable-scenarios` review found that claimed shared scenarios were persisted as real simulation runs, polluting metrics and making clones shareable.
- The `custom-withdrawal-strategies` review found premium withdrawal strategies exposed to the free tier.

The tests also rejected implementations. A recent FX change produced a gate run with three failures and 5,189 passes; PR #1099 then made the provider-backed tests hermetic. That is a real test-system catch, but it is test-infrastructure quality rather than a customer-facing product bug.

### Did those gates prevent later defects?

The repository can measure catches but cannot observe the counterfactual in which a rejected patch merged unchanged. Across the full-history cohort of 364 eligible product origins with mapped run state, the result shows strong selection bias:

| Earlier pre-merge result | Eligible origin PRs | Later defective origins |
|---|---:|---:|
| Reviewer did not catch | 93 | 5 (5.4%) |
| Reviewer caught | 271 | 32 (11.8%) |
| Tests did not catch | 218 | 18 (8.3%) |
| Tests caught | 146 | 19 (13.0%) |
| Neither caught | 74 | 4 (5.4%) |
| Either caught | 290 | 33 (11.4%) |

This does not mean review or tests cause bugs. Difficult and large PRs both trigger more pre-merge failures and create more opportunities for later escapes. The full-history test split now points in the same confounded direction as review; it provides no causal estimate of prevention.

The honest sentence for the post is: **independent review and tests stopped 65%-80% of implementation runs at least once, including concrete correctness and entitlement errors; the data cannot say how many production defects that prevented.**

## Current maintainability and tests

The current product-code language mix is 79,521 Python SLOC, 56,332 TypeScript, 18,245 CSS, and about 4,000 HTML/JSX/JavaScript. Tests contain 82,958 Python and 41,241 TypeScript SLOC.

The maintainability picture is reasonably good but not spotless:

- The former largest complexity hotspot, `ScenarioForm.tsx` at 6,871 code lines and complexity 1,237, is no longer present as one monolith.
- The current largest branch-complexity hotspot is `simulation_engine/core.py` at 5,347 code lines and complexity 725.
- `storage/repository.py` has grown from 7,374 to 14,497 `scc` code lines. Its measured branch complexity fell from 339 to 205, suggesting much of the growth is low-branch persistence plumbing, but it remains an obvious ownership and review hotspot.
- There are 68 product files with at least 500 code lines and 29 with at least 1,000.

The latest explicit full Python-suite count in the frozen PR data is 5,181 passed and eight skipped in PR #1091, plus frontend/Reddit tests and lint. Later PRs also passed their gates, but the cleaned logs do not retain a newer complete case count.

There is no configured statement or branch coverage instrument. The 78.6% test-to-product SLOC ratio is **not** test coverage. The defensible evidence is test volume, test accompaniment, green gates, and observed pre-merge failures.

## Models, review location, and infrastructure

Run state records the implementation agent family, not an exact model version. Across all 614 implementation-reaching runs, 460 (74.9%) used the Claude family and 154 (25.1%) used the Codex family.

| Implementation-reaching runs | Phase 1 | Phase 2 | Phase 3 |
|---|---:|---:|---:|
| Claude-family agent | 188 (74.0%) | 222 (82.5%) | 49 (54.4%) |
| Codex-family agent | 66 (26.0%) | 47 (17.5%) | 41 (45.6%) |
| Worktree backend | 254 (100%) | 269 (100%) | 12 (13.3%) |
| Container backend | 0 | 0 | 78 (86.7%) |

Commit co-author trailers provide the only exact implementation-model labels:

| Phase | Implementation-model evidence in commit trailers |
|---|---|
| Phase 1 | Predominantly Claude Opus 4.6 and Opus 4.6 (1M context); a few early Sonnet 4.6 mentions |
| Phase 2 | Claude Opus 4.7 (1M context), plus generic `Claude (Cowork)` trailers |
| Phase 3 | Opus 4.8, Sonnet 5, Fable 5, and Opus 5, plus generic `Claude (Cowork)` trailers |

The first later-model mentions are Opus 4.8 on June 18, Sonnet 5 on July 7, Fable 5 on July 8, and Opus 5 on July 29. These are commit-level provenance, not a one-model-per-run measurement: a squash commit can carry multiple labels, and generic Cowork trailers do not identify a model. The exact models used by Codex implementation runs cannot be recovered from the frozen state.

### Who reviewed, and where

The `review_caught` statistics in this report measure the independent **Codex review-decision gate**. They do not combine every review comment from every bot.

| Review-result source by implementation-reaching run | Phase 1 | Phase 2 | Phase 3 |
|---|---:|---:|---:|
| Local Codex review | 130 (51.2%) | 267 (99.3%) | 86 (95.6%) |
| Codex in GitHub Actions | 87 (34.3%) | 0 | 0 |
| Transition artifact with unknown source | 18 (7.1%) | 0 | 0 |
| No durable review-result artifact | 19 (7.5%) | 2 (0.7%) | 4 (4.4%) |

GitHub-hosted Codex review artifacts span March 3 through March 16. The first explicit local-review artifact is March 17. Local review ran `codex` CLI against a disposable worktree and published its structured decision back to the PR; later orchestrated PRs declared local ownership so the GitHub workflow did not call the cloud reviewer again.

The exact local Codex model is not durably recorded per run. A small number of Phase 1 and Phase 2 artifacts explicitly say `codex-gpt-5`; most say only `codex`. It would therefore be too strong to label all local review as GPT-5.

The GitHub path used `openai/codex-action@v1`. Its model floated and was not durably recorded per review until PR #963 pinned the fallback workflow to `gpt-5.5` on July 9. That pin followed the action default moving to unavailable `gpt-5.6-sol` and failing the gate. Because all explicit Phase 2 and Phase 3 review artifacts are local, the `gpt-5.5` pin does **not** identify the model behind those measured local reviews.

There was also a conditional Claude reviewer in GitHub Actions using `claude-sonnet-4-6`, focused on spec adherence, architecture, and test coverage. Its comments are auxiliary evidence; they are not the blocking decision source counted in the reviewer catch rates above.

Models, agent-family mix, infrastructure, product maturity, and work mix changed together. The three-phase trends describe the pipeline; they cannot isolate a model effect.

## Defect ledger: full history

All 1,016 PRs were reread under one semantic rubric. The 273 repair-candidate PRs represent 262 underlying engineering defects after related repair sequences were clustered and PRs that fixed independent failures were split:

- 175 strict bugs, 32 incomplete-feature defects, and 55 UX defects.
- 76 infrastructure defects and three test-system defects.
- Excluding those leaves 183 product/customer defects: 104 strict correctness bugs, 24 incomplete features, and 55 UX defects.
- Product/customer domains are 67 calculation/model, 39 product behavior, 20 data, 55 UX, and two security/privacy defects.

Examples include incorrect country tax policy, omitted capital gains in ACA MAGI, broken Google sign-in handoffs, stale solver state, contradictory MCP schemas, ignored positive savings, a Vietnam/US dual-tax crash, and local-currency amounts crossing a USD policy boundary without conversion.

### Repair workload

Repair share asks what each phase worked on. It does not assign the defect to the phase that introduced it.

| Fix/merge phase | Eligible product PRs | Strict product fix PRs | Broad product repair PRs |
|---|---:|---:|---:|
| Phase 1 | 177 | 34 (19.2%) | 59 (33.3%) |
| Phase 2 | 224 | 42 (18.8%) | 77 (34.4%) |
| Phase 3 | 103 | 20 (19.4%) | 43 (41.7%) |
| **All phases** | **504** | **96 (19.0%)** | **179 (35.5%)** |

By this measure, the strict bug-fix burden did **not** go down; it stayed near 19%. Broad repair work increased in Phase 3. That can coexist with better newer code because Phase 3 fixes include defects introduced by older work, and one repair PR can address more than one independent defect.

### Defect origins over time

Origin attribution uses explicit PR evidence first, then pre-fix changed-line blame, then the most recent prior product-file touch. An origin must have merged before the first fix in its repair sequence. Origins resolve for 222/262 engineering defects and 180/183 product/customer defects; attribution is still a proxy rather than proof of causality.

The raw lifetime view favors newer phases, but gives them less time to fail:

| Origin phase | Eligible origins | Median observation | Strict defective origins | Broad defective origins |
|---|---:|---:|---:|---:|
| Phase 1 | 177 | 147 days | 30 (17.0%) | 46 (26.0%) |
| Phase 2 | 224 | 105.5 days | 23 (10.3%) | 42 (18.8%) |
| Phase 3 | 103 | 16 days | 7 (6.8%) | 14 (13.6%) |

A fixed 30-day window makes the comparison fairer by including only origins that had 30 days to reveal a defect and counting only defects first repaired within those 30 days:

| Origin phase | 30-day-mature origins | Strict defective origins | Broad defective origins |
|---|---:|---:|---:|
| Phase 1 | 177 | 24 (13.6%) | 41 (23.2%) |
| Phase 2 | 224 | 19 (8.5%) | 36 (16.1%) |
| Phase 3 | 43 | 3 (7.0%) | 6 (14.0%) |

The 14-day strict rates show the same direction: 12.4%, 7.1%, and 5.0% across 177, 224, and 60 mature origins. The honest conclusion is that **the observed defect-introduction rate went down**, including under equal exposure, while the amount of repair work did not. Phase 3's mature samples are still small, and model, workflow, infrastructure, work mix, and product maturity changed together.

### Origin change size

| Origin PR churn | Origin PRs | Strict-defective PRs | Defective rate | Strict defects/K changed lines |
|---|---:|---:|---:|---:|
| 0-49 | 22 | 0 | 0.0% | 0.000 |
| 50-199 | 76 | 1 | 1.3% | 0.109 |
| 200-499 | 118 | 9 | 7.6% | 0.213 |
| 500-999 | 119 | 17 | 14.3% | 0.246 |
| 1,000-1,999 | 118 | 17 | 14.4% | 0.182 |
| 2,000+ | 51 | 16 | 31.4% | 0.198 |

The chance that a PR is later implicated rises sharply with change size: 31.4% for 2,000+ lines versus 9.7% below 2,000. Defects per changed KLOC do not rise in the same way because the denominator scales with PR size. The actionable result is about review and failure surface per change: split the largest specs and PRs.

### Origin work mode

| Origin provenance | Eligible origins | Broad-defective origins | Strict-defective origins | Broad defects/K changed | Strict defects/K changed |
|---|---:|---:|---:|---:|---:|
| `spec implement` | 279 | 59 (21.1%) | 32 (11.5%) | 0.265 | 0.143 |
| `spec task` | 85 | 10 (11.8%) | 5 (5.9%) | 0.318 | 0.139 |
| Other code PR | 140 | 33 (23.6%) | 23 (16.4%) | 0.681 | 0.443 |

Tasks have the lowest defective-origin rate, but they are much smaller changes. Their strict defect density per changed KLOC is almost identical to full specs (0.139 versus 0.143). The higher rate for other code PRs is also confounded: this bucket contains substantial early work whose run provenance was not durably mapped, plus ad hoc corrective work. It does not mean manually authored code.

### Density

Two retrospective ratios are useful only as scale checks:

- `104 / 187.935 current non-test KSLOC = 0.55` known repaired strict product defects per KLOC.
- `183 / 187.935 = 0.97` known repaired broad product/customer defects per KLOC.

The earlier 0.74 figure combined 110 defects from the old cumulative rubric with 29 from the later rubric. It is retired. The consistent full-history ledger produces 0.55 strict and 0.97 broad, which explains why earlier summaries appeared close to one defect/KLOC depending on the definition. Neither is a remaining-defect density: the numerator is historical repaired defects and the denominator is the current tree after deletion and replacement.

## Comparison with traditional development

There is no clean manual-team control group. Commits/day, defects/KLOC, corrective-work share, and review catches all measure different constructs. The useful comparison is a set of reference points, not a single league table.

### Corrective-work share

The [Corrective Commit Probability study](https://arxiv.org/abs/2007.10912) examined 7,557 active GitHub projects. Its valid-project distribution had a 20% median corrective-commit probability, 6% at the high-quality 90th percentile, and 39% at the low-quality 10th percentile.

Retirement Lab's full-history 19.0% strict product-fix PR share is close to that 20% median. The 35.5% broad share, which includes incomplete and UX follow-ups, is close to the study's 39% low-quality decile. Neither comparison is apples-to-apples: this ledger uses semantic PR descriptions and a product-only denominator, while CCP uses a linguistic commit classifier. The fair conclusion is that the strict repair burden looks ordinary, while broad rework remains substantial.

A [manually labeled study of 3,498 bug-fixing commits](https://link.springer.com/article/10.1007/s10664-021-10083-5) found that only 25.1% of changed lines in those commits were actual bug-fix lines; tests were 39.6% and documentation 14.0%. That is why this report classifies PR intent and defects instead of treating every line in a fix PR as a defect.

### Defect density

A [scoping study of 109 projects](https://iris.polito.it/handle/11583/2502546) reported a median 4.3 and mean 7.47 post-release defects/KLOC, with substantial variation between projects. Retirement Lab's full-history PR ledger contains 0.55 known repaired strict defects per current KLOC, or 0.97 when incomplete-feature and UX defects are included. The definitions and observation processes are not identical, so these figures do not establish that Retirement Lab is four to eight times better. They are nevertheless encouraging: all known repair work flows through the repository, the broader measure includes more than conventional correctness bugs, and the observed density remains well below the study's median.

Retirement Lab is young, so additional defects will be discovered and its recent work is especially right-censored. At its current size, however, the broad ledger would have to grow from 183 to roughly 808 defects to reach the study median. The current-tree denominator also does not perfectly match the historical defect numerator because code has been added, replaced, and deleted. The reasonable conclusion is not superiority, but that the quality observed so far is numerically favorable and provides legitimate cause for cautious optimism.

### Change size and reviewability

Google's [small-change guidance](https://google.github.io/eng-practices/review/developer/small-cls.html) says 100 lines is usually reasonable and 1,000 lines is usually too large; smaller changes are reviewed more thoroughly and are less likely to introduce bugs.

The repository's own outcome data agrees with the direction of that guidance: full-history origin PRs with 2,000+ changed lines had a 31.4% strict-defective rate, versus 9.7% below 2,000. Phase 3 also has the largest share of 2,000+ commits at 13.1%. This is a stronger and more honest comparison than claiming an abstract code-quality score.

### Testing practice

The same Google guidance says tests are expected for changes that add or alter logic. Retirement Lab improved from 80.4% product-change accompaniment in Phase 1 to more than 91% in Phases 2 and 3. Full-spec PRs were almost always accompanied by tests; tasks were the main Phase 1/2 gap. Because no coverage gate is configured, the repository cannot be compared on branch or statement coverage.

### Throughput and shipped value

A 2026 [NBER study of more than 100,000 GitHub developers](https://www.nber.org/papers/w35275) found cumulative effects of autonomous agents of roughly +180% on commits, but only +50% on projects and +30% on releases. That is a useful warning against turning Retirement Lab's commits/day into a manual-team equivalence claim. The live product is the best evidence that the work shipped; release, adoption, reliability, and business outcomes are better follow-on measures than commit volume.

External AI evidence also depends heavily on workflow. METR's [early-2025 randomized trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) found 16 experienced open-source developers took 19% longer with then-current AI tools across 246 issues. That setting used familiar developers doing two-hour issues with interactive tools; Retirement Lab uses a non-coding operator and long autonomous loops with independent machine QA. The results are not contradictory, but they are not directly comparable.

DORA is intentionally absent. The repository snapshot does not provide a trustworthy deployment-frequency, lead-time, failed-deployment, and restoration ledger, and those measures would not answer the narrower question of defect quality anyway.

## What the data supports

1. **A non-coding operator can sustain a real software product.** The current tree has 187,935 non-test programming-language lines and 124,199 test lines, and the live product is independently inspectable.
2. **Most orchestrated implementations complete without a keyboard return.** The primary mapped-run estimate improves from 81.9% to 90.7% across the three phases.
3. **The workflow is not unreviewed generation.** Review or tests rejected 65%-80% of implementations at least once, including concrete correctness, concurrency, entitlement, and data-integrity errors.
4. **Quality did not obviously collapse as the codebase grew.** Test accompaniment improved, test volume grew, aggregate complexity density fell, and the former worst hotspot was decomposed.
5. **Observed defect introduction improved across the three phases.** The 30-day strict defective-origin rate fell from 13.6% to 8.5% to 7.0%, though Phase 3 has only 43 mature origins.
6. **Agent-written software still has ordinary software risks.** The full-history audit found 104 repaired strict product defects; broad repair work consumed 35.5% of eligible product PRs.
7. **The workflow's biggest measured risk is oversized changes.** Defect concentration above 2,000 changed lines is too large to ignore, and those changes became more common in Phase 3.

## What it does not support

- That manual coding is dead for every developer, codebase, or organization.
- That Retirement Lab has better quality than a strong manual team.
- That 65%-80% of potential production defects were prevented by the reviewer/tests.
- That newer models caused the phase improvements; model, infrastructure, work mix, and project age changed together.
- That 3.0 or 13.6 commits/day equals a particular number of human engineers.
- That test SLOC or declaration count is coverage.
- That Phase 3's current 30-day result is a settled long-run rate; only 43 origins are mature enough for it.

## Recommended post thesis

> I stopped writing code and started operating a software factory. The work arrived in three bursts: greenfield construction, productization, then platform expansion and hardening. By the third phase, 91% of merged implementation runs showed no evidence that I had to return to the keyboard after starting them, even though independent review or tests rejected 80% at least once. The result was a live product with 188,000 non-test programming-language lines and 124,000 test lines. It was not flawless: a full-history audit found 104 repaired strict product defects. The 30-day defective-origin rate fell across the three phases, but one in three eligible product PRs was still repair or follow-up work, and the largest changes were much more likely to need a later fix. Implementation became a supervised, measurable pipeline rather than my primary job.

Possible title: **I stopped coding. Then I measured what the agents shipped.**

The most credible ending is not "agents are better than teams." It is: **this is already a viable way for one person to produce and maintain serious software, provided the process treats review, tests, telemetry, defect follow-up, and change-size control as first-class work.**

## Suggested article structure

1. **What changed after "manual coding is dead."** No file editing; goals, specs, launch decisions, and exception handling became the human work.
2. **What the agents actually built.** Link the live product and describe its calculation, tax, data, API/MCP, frontend, mobile, auth, billing, analytics, and operational surface.
3. **The three development phases.** Show the two natural pauses and explain the shift from bootstrap to productization to platform/hardening.
4. **How the pipeline works.** Name the Claude/Codex implementation families, explain the GitHub-to-local Codex review transition, then show verify gates, retries, and merge.
5. **Did I have to intervene?** Present the direct 81.9% to 90.7% trend and the broad sensitivity result, including exact definitions.
6. **The quality filter is doing real work.** Present the 65%-80% pre-merge catch rates and concrete reviewer findings.
7. **The bugs that still escaped.** Present the 104 full-history strict product defects, flat 19% strict repair share, declining 30-day origin rate, and calculation/auth/MCP examples.
8. **What looks like a normal team and what does not.** Use corrective-share and defect-density reference points with caveats; do not convert commits into headcount.
9. **What I would improve next.** Split large specs/PRs, instrument coverage, keep a production incident ledger, and track deployments, releases, and customer-visible outcomes.
10. **Conclusion.** Manual implementation is optional in this case; engineering judgment and quality systems are not.

## Next measurements worth adding

- Statement and branch coverage, frozen at every release.
- Deployment and rollback ledger tied to PRs.
- Production exceptions, incidents, and customer-reported defects with first-seen and fixed dates.
- Spec-start to merge lead time and agent compute cost per merged PR.
- Refresh the 30-day cohort and add 60/90-day horizons as Phase 3 matures.
- PR-size policy and before/after defect rate once 2,000+ line changes are split.
- Release/adoption outcomes so throughput can be discussed as shipped value rather than commit activity.

## Artifact map

- `data/development-phases-consolidated-2026-08-14.json`: frozen three-phase throughput, run, QA, intervention, mode, stall, and endpoint code metrics.
- `data/development-phase-endpoints-2026-08-14.json`: exact code-tree and test-declaration inputs at PRs #520, #911, and #1101.
- `scripts/analyze_development_phases.py`: reproducible phase aggregation from the frozen repository, PR, run, and endpoint data.
- `data/run-metrics-consolidated-2026-08-14.json`: intervention, stalls, reviewer, tests, modes, agents, and per-run rows.
- `data/repository-consolidated-2026-08-14.json`: commit/day, churn, test accompaniment, and per-commit rows.
- `data/pr-metrics-full-history-2026-08-14.json`: PR size, file categories, run mapping, and one-rubric semantic class for all 1,016 PRs.
- `data/defect-items-full-history-2026-08-14.json`: 262 reconciled underlying defects represented by 273 repair PRs.
- `data/defect-ledger-full-history-2026-08-14.json`: full-history origins, phase rates, fixed observation windows, size rates, and work-mode rates.
- `data/repair-rates-full-history-2026-08-14.json`: strict and broad repair workload by phase and PR provenance.
- `data/review-test-escape-full-history-2026-08-14.json`: full-history non-causal catch/escape correlation.
- `data/code-metrics-consolidated-2026-08-14.json`: same-method SLOC and complexity at PRs #797 and #1101.
- `data/code-hotspots-consolidated-2026-08-14.json`: file-size and complexity hotspots.
- `data/test-inventory-consolidated-2026-08-14.json`: static test declarations at PRs #797 and #1101.
- `data/model-evidence-consolidated-2026-08-14.json`: implementation families, trailer model labels, review location by phase, and reviewer-model evidence.
- `methodology/defect-classification-rubric.md`: semantic classification rules.

Important implementation audit corrections retained during consolidation:

- Origins in a multi-PR repair sequence must predate the first fix PR; an earlier PR in the same repair sequence cannot be called the introducing origin.
- Older GitHub Actions job IDs are not parsed as PR numbers. Merged-run metrics join through the explicit run ID in the PR ledger or an actual `/pull/<number>` URL.