This agent-authored report accompanies Thiago Hirai’s Manual Code Review Is Optional. It analyzes the repository, pull requests and execution logs behind the essay. The evidence archive contains selected original specs, review findings, tests and repairs.

The analysis covers February 23-August 14, 2026. Publication on September 10 does not extend that observation period. The notes below explain the essay’s headline figures; the original three-phase report follows.

Notes on the Essay’s Figures

Throughput

42 PRs/week is 851 code-changing PRs divided by 141 calendar days in the three development periods, multiplied by seven: 42.25. It excludes 159 spec-authoring-only PRs and six documentation/content PRs.

The calculation excludes two long breaks with no first-parent commits: April 4-21 (18 days) and June 4-17 (14 days). It includes weekends and shorter inactive stretches within each development period. Across all 173 calendar days, including both breaks, the corresponding rate is 34.43 code PRs/week.

This measures repository throughput, not PRs per hour of Thiago’s work. The records do not measure his working hours or establish a productivity multiplier over a human team.

Intervention

410 out of 478 specs (85.8%) refers to completed implementation runs mapped to merged PRs: 321 spec implement runs and 157 spec task runs. Both modes execute committed specifications; tasks are generally smaller.

The 410 had no recorded direct intervention signal after implementation began. The primary proxy counts recorded operator responses, retry-cap increases when the run crossed its original cap, and recovery from a non-retriable failed state. It includes superseded ancestor runs so a restart cannot erase earlier intervention. This is a log-derived estimate, not keyboard telemetry, and its denominator is completed mapped runs rather than every attempted spec.

A broader sensitivity check adds explicit kill markers and stalls of at least two hours; 389/478 (81.4%) had no signal under that definition. Automated revisions do not count as human intervention: 275 of the 410 runs were rejected by the reviewer at least once before finishing.

Repair Share

About one in five product PRs means 96 of 504 eligible product PRs (19.0%) addressed strict correctness bugs. The Corrective Commit Probability study, based on 7,557 active GitHub projects, reported a median estimated bug-fixing commit share of 20% among valid projects.

Both measures concern bug-fixing work, making the study a useful reference point. They are not identical: the study classifies commit messages, while this analysis classifies PR descriptions and uses a product-only denominator. The study does not apply our exact classification boundaries.

Our strict figure excludes repairs classified as UX defects or incomplete features. The UX category covers behavior that worked mechanically but was materially misleading, confusing, inaccessible or unusable; a UI failure that breaks expected functionality can still qualify as a strict bug. Including the additional repair categories raises the share to 179/504 (35.5%). These are shares of repair work, not percentages of changes that introduced defects; a repair may address an older problem or several defects.

Examples and Source Records

• A waiting request holds a database connection: the original review finding, merged correction and regression test.
• Other independent review findings: consent ordering and duplicate simulations.
• Testing layout invariants: the overlap instruction, test and later extensions.
• The currency error and repair: withdrawal and repair PR descriptions, with merge timestamps.
• Essay metrics (JSON) and methodology.

Snapshot Corrections

A later boundary check counted 1,077 first-parent commits, five more than the original report’s 1,072 because its date filter omitted five commits on the first day. The original commit tables below retain that earlier count; the essay’s throughput calculation uses merged PRs and is unaffected. The full-spec median is 878 changed lines; 860 is the size of the selected clean-spec example, not the median.

The original Markdown report remains available unchanged.

---

Generated 2026-08-14 (America/Los_Angeles).

Executive read

Retirement Lab is strong evidence for a narrow claim: a single operator writing no source code can build and maintain a substantial live product by specifying work and supervising agents. It is not evidence that agents produce defect-free code, that manual coding is obsolete in every setting, or that this workflow beats a strong conventional team on quality.

The history is best understood as three active development phases, separated by the repository’s only two double-digit commit pauses:

1. Bootstrap and core construction, February 23-April 3: 545 first-parent commits in 40 calendar days.
2. Productization and UX build-out, April 22-June 3: 351 commits in 43 days.
3. Platform expansion and hardening, June 18-August 14: 176 commits in 58 days.

The most defensible results are:

• Throughput stepped down from 13.6 to 8.2 to 3.0 commits per calendar day, and from 15.1 to 9.2 to 5.2 per active day. The project did not sustain its greenfield pace.
• The mapped merged-run no-direct-intervention rate improved from 81.9% to 86.0% to 90.7%. This is the best repository-derived proxy for not returning to the keyboard after implementation started.
• Independent review or tests rejected 65.4%, 68.0%, and 80.0% of implementation-reaching runs at least once across the three phases. The agents were not simply generating code and merging it unchecked.
• Implementation used 460 Claude-family and 154 Codex-family runs. The measured blocking reviewer was a separate Codex gate: GitHub Actions early in Phase 1, then predominantly local Codex CLI review from March 17 onward.
• At each phase endpoint, product SLOC was 45,310, 119,342, and 158,063; test SLOC was 37,340, 83,513, and 124,199. Product complexity density fell from 115.4 to 96.5 to 90.4 scc branches per KLOC.
• Product-change test accompaniment rose from 80.4% in Phase 1 to 91.7% in Phase 2 and held at 91.3% in Phase 3. There is still no configured statement or branch coverage instrument.
• Change size became more concerning as throughput fell. The share of commits with at least 2,000 changed lines rose from 4.6% to 6.8% to 13.1%.
• A consistent semantic review of all 1,016 merged PRs found 262 underlying engineering defects addressed by 273 repair PRs. Excluding infrastructure and test-system failures leaves 183 product/customer defects, including 104 strict correctness bugs.
• Strict product repair work stayed essentially flat at 19.2%, 18.8%, and 19.4% of eligible product PRs. Origin attribution tells a different story: the 30-day strict defective-origin rate fell from 13.6% to 8.5% to 7.0%. Phase 3 has only 43 origins old enough for that comparison, so this is encouraging rather than conclusive.
• Change size is the clearest measured quality risk. Sixteen of 51 eligible origin PRs with 2,000+ changed lines later revealed a strict product defect (31.4%), versus 44 of 453 smaller PRs (9.7%).

The best article is therefore an instrumented case study about a different division of labor: the human chooses and specifies work; implementation, verification, review, and repair are mostly machine-operated. The bugs did not disappear. The keyboard work moved up a level.

Scope, cutoff, and phase boundaries

• Git and PR cutoff: origin/master at 2cdd7383, merged PR #1101 on 2026-08-14 at 07:27 Pacific.
• Git history: 1,072 first-parent commits from February 23 through the cutoff.
• Run telemetry: 711 deduplicated run records from the Mac and thiago-linux; 614 reached implementation, 599 passed, and 478 map to merged PRs.
• Phase run tables assign by local run creation date. One implementation-reaching run began on June 17 during the second commit pause and is outside the three phase totals, although its task PR #912 merged on June 18.
• Semantic review: all 1,016 merged PRs were classified from title, description, and changed files under one rubric. The 273 repair candidates were clustered into 262 underlying defects, or split when one PR fixed independent failures.
• Code-tree comparison: exact detached trees at PRs #520, #911, and #1101, measured with the same cloc, scc, and test-declaration rules.

Phase	Active dates	Endpoint	Characterization
1. Bootstrap and core construction	Feb 23-Apr 3	PR #520	Core simulator, tax logic, initial product, testing, and agent workflow
Pause	Apr 4-Apr 21	18 inactive days	No first-parent commits
2. Productization and UX build-out	Apr 22-Jun 3	PR #911	Design system, scenario UX, auth, observability, relocation, sharing, and mobile planning
Pause	Jun 4-Jun 17	14 inactive days	No first-parent commits
3. Platform expansion and hardening	Jun 18-Aug 14	PR #1101	Public MCP, mobile platform, distribution, country/tax expansion, currency correctness, and product hardening

These boundaries come from the work cadence, not a desired story. The 18-day and 14-day pauses are the only double-digit inactive gaps; the next-largest gap is five days. Mid-March QA adoption, the May 12 snapshot, and the July container migration are meaningful workflow events inside phases, but none is as strong a top-level development boundary.

The May snapshot represents the project from inception through PR #797, not the month of May. The defect ledger no longer depends on that old boundary: the pre-#798 PRs were reread under the same rubric as the later PRs. Treating May 12 as a development phase boundary would still mix a historical data-collection boundary with the shape of the work.

Thiago’s zero-manual-coding statement is a self-report. Git cannot prove that a person never edited a file. The run ledger can independently show how often the orchestrated implementation path contains evidence that the operator had to return to the keyboard.

Workflow being measured

The recurring production path is:

1. The operator selects a goal and creates or commits a spec.
2. An implementation agent works in an isolated worktree or container.
3. The verify phase runs tests and lint independently of the implementation conversation.
4. An independent code reviewer can request changes.
5. Findings are fed back to the implementation agent until the gates pass or the run exhausts its retry policy.
6. The system publishes and merges the PR.
7. Later production, QA, or analysis findings become new specs, creating the defect evidence in this report.

This is closer to operating a software-production system than to using autocomplete. The relevant unit is the complete run and its quality gates, not the number of lines an LLM emitted.

Three-phase scorecard

Delivery shape

Metric	Phase 1	Phase 2	Phase 3
Calendar days in active phase	40	43	58
First-parent commits	545	351	176
Commits/calendar day	13.62	8.16	3.03
Active commit days	36	38	34
Commits/active day	15.14	9.24	5.18
Median commit churn	175	467	395
P90 commit churn	1,338	1,603	2,183
Commits with 2,000+ changed lines	25 (4.6%)	24 (6.8%)	23 (13.1%)
Product lines added	134,974	140,649	64,099
Test lines added	77,879	73,900	52,919
Test additions/product additions	0.577	0.525	0.826

Phase 1 contains 50.8% of all first-parent commits, Phase 2 contains 32.7%, and Phase 3 contains 16.4%. Lower later throughput is not itself a failure: the product was larger, work shifted toward wider platform changes, and inactive days became more common. But commits/day cannot support a claim that one operator equals a fixed number of engineers.

The change-size distribution is less comfortable. Median churn peaks in Phase 2, while the upper tail keeps growing through Phase 3. The later defect attribution strongly suggests that the 2,000+ line tail is a process risk, regardless of who or what wrote the code.

Autonomy and machine QA

Metric	Phase 1	Phase 2	Phase 3
Implementation-reaching runs	254	269	90
Passed runs	248 (97.6%)	264 (98.1%)	86 (95.6%)
Runs mapped to merged PRs	127	264	86
Mapped runs with no direct intervention	104 (81.9%)	227 (86.0%)	78 (90.7%)
Mapped runs with no broad intervention	99 (78.0%)	219 (83.0%)	70 (81.4%)
Reviewer requested changes	143 (56.3%)	162 (60.2%)	68 (75.6%)
Test gate failed at least once	109 (42.9%)	100 (37.2%)	20 (22.2%)
Reviewer or tests caught something	166 (65.4%)	183 (68.0%)	72 (80.0%)
Product commits accompanied by tests	315/392 (80.4%)	278/303 (91.7%)	116/127 (91.3%)

The direct autonomy result improves monotonically. The broad result does not: Phase 3 falls from 83.0% to 81.4% because the broad definition treats two-hour durable gaps as possible intervention, and July’s container migration produced a temporary stall cluster. That is why the direct proxy is the primary keyboard measure and the broad proxy is a pessimistic sensitivity bound.

The QA mix also changes. Review catches rise sharply while test-gate catches fall. Their union reaches 80.0% in Phase 3, so the pipeline was more likely to reject an implementation at least once, not less. The repository cannot establish whether that reflects stronger review, harder full-spec work, fewer flaky test failures, model changes, or all of them.

Code at each endpoint

Same-method tree metric	PR #520	PR #911	PR #1101
Product SLOC	45,310	119,342	158,063
Support/tool SLOC	6,021	25,600	29,872
Test SLOC	37,340	83,513	124,199
Test/product SLOC ratio	82.4%	70.0%	78.6%
Static test declarations	1,818	3,774	5,770
Product files (cloc)	174	381	528
Product complexity/KLOC (scc)	115.4	96.5	90.4

The current tree has 187,935 non-test programming-language lines when product and support/tool categories are combined. Test volume did not grow in a straight line relative to product code, but the final ratio recovered to 78.6%, static test declarations more than tripled from the Phase 1 endpoint, and aggregate product complexity density fell 21.7%.

These are encouraging maintainability signals, not proof of maintainability. Generated volume can lower aggregate density, and scc complexity does not measure architectural coupling, readability, or correctness.

What changed in each phase

Phase 1: bootstrap and core construction

This was the greenfield burst: core simulation and tax behavior, inputs and results, shareable scenarios, an initial business surface, Playwright, linting, make targets, and dual AI review. Of 489 PRs, 274 were unmapped code changes, 87 were spec-authoring commits, 70 were full-spec implementations, and 57 were tasks. The relatively low 80.4% test accompaniment and 56.3% review-catch rate include the period when the workflow itself was still being assembled.

The phase produced 45,310 product SLOC and 37,340 test SLOC by PR #520. Its 13.6 commits/day is real repository activity, but it is primarily a bootstrap rate, not a sustainable mature-product baseline.

Phase 2: productization and UX build-out

The second burst shifts from bootstrapping to deliberate product orchestration. The PR mix contains 170 full-spec implementations, 93 tasks, 41 spec-authoring commits, and only 45 unmapped code changes. Major work includes the design system and scenario redesign, authentication and observability, relocation analysis, glide paths, sharing, and mobile planning.

Product SLOC grew by 74,032 between endpoints, while product complexity density fell from 115.4 to 96.5 per KLOC. Product-change test accompaniment reached 91.7%. Throughput was lower than Phase 1, and it declined within the phase, but there is no repository pause or categorical workflow shift exactly at the May 12 snapshot.

Phase 3: platform expansion and hardening

The third burst adds public MCP, currency contracts and sharing, mobile shell/IAP/push, Reddit distribution, country rankings and content, France-US and Vietnam tax work, FX correctness, Pro gates, and analytics. Of 176 PRs, 81 were full-spec implementations, 31 authored specs, 54 were unmapped changes, seven were tasks, and three were docs/content.

This is the slowest phase and the one with the fattest change-size tail. It is also the strongest phase on direct autonomy, machine-QA rejection, test additions relative to product additions, and aggregate complexity density. Both can be true: the pipeline looks more capable while its largest changes look riskier.

Within Phase 3, model labels begin changing on June 18 and execution begins moving to Linux containers at the end of June. July’s stall spike and later recovery are operational subphases, not evidence for a fourth product-development phase.

Intervention: when the operator had to return

“Intervention” means a keyboard action after implementation started. No single repository field records that fact, so the analysis uses two auditable proxies.

The direct proxy counts:

• an explicit operator response recorded during the run;
• a retry cap increased beyond the cap in force when the run began, with the run actually crossing the old cap; or
• a failed state, which is not retriable without a person restarting or replacing the run.

The broad proxy adds:

• an explicit operator-kill marker; or
• a gap of at least two hours between one durable phase event ending and the next beginning.

A long gap is not proof that a key was pressed, which is why it is a sensitivity bound rather than part of the primary metric. For a merged run, both scores include superseded ancestor runs. Restarting a failed implementation under a new run ID therefore cannot erase earlier evidence.

Direct proxy, mapped merged runs	Phase 1	Phase 2	Phase 3
No intervention signal	104/127 (81.9%)	227/264 (86.0%)	78/86 (90.7%)
Exactly one signal	1/127 (0.8%)	17/264 (6.4%)	4/86 (4.7%)
Two or more signals	22/127 (17.3%)	20/264 (7.6%)	4/86 (4.7%)

Broad proxy, mapped merged runs	Phase 1	Phase 2	Phase 3
No intervention signal	99/127 (78.0%)	219/264 (83.0%)	70/86 (81.4%)
Exactly one signal	6/127 (4.7%)	14/264 (5.3%)	9/86 (10.5%)
Two or more signals	22/127 (17.3%)	31/264 (11.7%)	7/86 (8.1%)

Across implementation-reaching runs, the intervention signals were:

Signal	Phase 1	Phase 2	Phase 3
Explicit operator response	1	30	4
Retry-cap increase after crossing inferred initial cap	32	13	4
Failed-state recovery	5	5	4
Explicit manual-kill marker	0	1	0

Signals overlap, so they do not sum to intervention-bearing runs. The explicit kill marker contributes only to the broad proxy.

Stall sensitivity was:

Maximum observable durable gap	Phase 1	Phase 2	Phase 3
At least 1 hour	37/254 (14.6%)	40/269 (14.9%)	15/90 (16.7%)
At least 2 hours	29/254 (11.4%)	33/269 (12.3%)	13/90 (14.4%)
At least 4 hours	23/254 (9.1%)	26/269 (9.7%)	12/90 (13.3%)
At least 8 hours	21/254 (8.3%)	21/269 (7.8%)	8/90 (8.9%)

July is the important outlier within Phase 3: 12/41 July runs had a two-hour gap and eight had an eight-hour gap. By August those figures were 1/38 and 0/38. The direct metric says autonomy improved; the broad metric correctly warns that the container transition temporarily made runs look less continuously unattended.

Spec-managed work versus other PRs

The workflow provenance ledger distinguishes committed full-spec implementation, committed task implementation, spec authoring, and PRs that cannot be durably mapped to either committed-spec run mode. “Other code” does not mean manually written; it means the PR lacks a durable spec implement or spec task mapping.

PR provenance	Phase 1	Phase 2	Phase 3	All phases
spec implement	70	170	81	321 (31.6%)
spec task	57	93	7	157 (15.5%)
Other code PR	274	45	54	373 (36.7%)
Spec/task authoring only	87	41	31	159 (15.6%)
Docs/content	1	2	3	6 (0.6%)
All merged PRs	489	351	176	1,016

The workflow changed materially. Only 26.0% of Phase 1 PRs were mapped implementations of a committed spec or task, versus 74.9% in Phase 2 and 50.0% in Phase 3. Phase 1 therefore mixes product bootstrap with substantial pre-orchestrator or unmapped agent work. Phase 2 is the cleanest measure of the committed-spec pipeline. Phase 3 returns to more unmapped platform, content, and corrective work while almost eliminating short tasks.

All-history PR shape	PRs	Median churn	Product PRs accompanied by tests
spec implement	321	878	315/317 (99.4%)
spec task	157	242	125/155 (80.6%)
Other code PR	373	174	260/307 (84.7%)
Spec/task authoring only	159	164	0/4
Docs/content	6	242	n/a

Full specs are a distinct unit: 3.6x larger than tasks and 5.0x larger than other code PRs by median churn. They also have nearly universal test accompaniment. Intervention and reviewer-cycle comparisons are available only for mapped spec/task runs; assigning those run outcomes to the 373 other code PRs would fabricate telemetry that does not exist.

spec task versus spec implement

spec implement consumes a committed specs/*.md file. spec task consumes a smaller committed specs/tasks/*.md instruction. They are different work units, not interchangeable prompt styles.

Work size and testing

Metric	P1 task	P1 spec	P2 task	P2 spec	P3 task	P3 spec
Implementation PRs	57	70	93	170	7	81
Median total PR churn	195	1,097	299	851	279	958
Median product churn	75	449	111	452	55	479
Product PRs accompanied by tests	78.9%	100.0%	80.4%	98.8%	100.0%	100.0%

Full-spec implementations were 2.8x to 5.6x larger by median total churn. The task path stayed much smaller and was the persistent testing gap in Phases 1 and 2. By Phase 3 there are only seven task PRs, so its 100% accompaniment rate is not a stable trend.

Run outcomes

Metric	P1 task	P1 spec	P2 task	P2 spec	P3 task	P3 spec
Implementation-reaching runs	74	179	93	176	6	84
Pass rate	100.0%	96.6%	100.0%	97.2%	100.0%	95.2%
Reviewer or tests caught something	78.4%	59.8%	55.9%	74.4%	16.7%	84.5%
Mapped runs with no direct intervention	89.5%	75.7%	96.8%	80.1%	100.0%	90.0%
Mapped runs with no broad intervention	86.0%	71.4%	93.5%	77.2%	83.3%	81.2%

The Phase 1 catch-rate reversal is a warning against treating mode as a causal variable: workflow maturity and task selection differed. In Phases 2 and 3, full specs are much more likely to be challenged by QA, which is consistent with their size and complexity. The six Phase 3 task runs are too few for a useful comparison.

Independent reviewer and tests

Across all 614 runs that reached implementation, the independent reviewer requested changes at least once in 373 (60.7%, rounded to 61% in the essay). Each run is counted once, regardless of how many review cycles it required. This denominator includes all implementation-reaching runs, not just the 478 mapped to merged PRs. One run began during the June pause and is included in this overall figure but falls outside the three phase columns below.

Implementation-reaching runs	Phase 1	Phase 2	Phase 3
Reviewer requested changes	143/254 (56.3%)	162/269 (60.2%)	68/90 (75.6%)
Test/verify gate failed	109/254 (42.9%)	100/269 (37.2%)	20/90 (22.2%)
Either mechanism rejected the run	166/254 (65.4%)	183/269 (68.0%)	72/90 (80.0%)

Examples show that the reviewer was finding consequential issues, not just formatting:

• The auth-conversion-gates review found that the anonymous run cap used a non-atomic check/increment and could be bypassed with concurrent requests.
• The form-people-model review found that migrated couple scenarios could still be saved without required partner fields and that Pro-only controls leaked outside the gate.
• The shareable-scenarios review found that claimed shared scenarios were persisted as real simulation runs, polluting metrics and making clones shareable.
• The custom-withdrawal-strategies review found premium withdrawal strategies exposed to the free tier.

The tests also rejected implementations. A recent FX change produced a gate run with three failures and 5,189 passes; PR #1099 then made the provider-backed tests hermetic. That is a real test-system catch, but it is test-infrastructure quality rather than a customer-facing product bug.

Did those gates prevent later defects?

The repository can measure catches but cannot observe the counterfactual in which a rejected patch merged unchanged. Across the full-history cohort of 364 eligible product origins with mapped run state, the result shows strong selection bias:

Earlier pre-merge result	Eligible origin PRs	Later defective origins
Reviewer did not catch	93	5 (5.4%)
Reviewer caught	271	32 (11.8%)
Tests did not catch	218	18 (8.3%)
Tests caught	146	19 (13.0%)
Neither caught	74	4 (5.4%)
Either caught	290	33 (11.4%)

This does not mean review or tests cause bugs. Difficult and large PRs both trigger more pre-merge failures and create more opportunities for later escapes. The full-history test split now points in the same confounded direction as review; it provides no causal estimate of prevention.

The honest sentence for the post is: independent review and tests stopped 65%-80% of implementation runs at least once, including concrete correctness and entitlement errors; the data cannot say how many production defects that prevented.

Current maintainability and tests

The current product-code language mix is 79,521 Python SLOC, 56,332 TypeScript, 18,245 CSS, and about 4,000 HTML/JSX/JavaScript. Tests contain 82,958 Python and 41,241 TypeScript SLOC.

The maintainability picture is reasonably good but not spotless:

• The former largest complexity hotspot, ScenarioForm.tsx at 6,871 code lines and complexity 1,237, is no longer present as one monolith.
• The current largest branch-complexity hotspot is simulation_engine/core.py at 5,347 code lines and complexity 725.
• storage/repository.py has grown from 7,374 to 14,497 scc code lines. Its measured branch complexity fell from 339 to 205, suggesting much of the growth is low-branch persistence plumbing, but it remains an obvious ownership and review hotspot.
• There are 68 product files with at least 500 code lines and 29 with at least 1,000.

The latest explicit full Python-suite count in the frozen PR data is 5,181 passed and eight skipped in PR #1091, plus frontend/Reddit tests and lint. Later PRs also passed their gates, but the cleaned logs do not retain a newer complete case count.

There is no configured statement or branch coverage instrument. The 78.6% test-to-product SLOC ratio is not test coverage. The defensible evidence is test volume, test accompaniment, green gates, and observed pre-merge failures.

Models, review location, and infrastructure

Run state records the implementation agent family, not an exact model version. Across all 614 implementation-reaching runs, 460 (74.9%) used the Claude family and 154 (25.1%) used the Codex family.

Implementation-reaching runs	Phase 1	Phase 2	Phase 3
Claude-family agent	188 (74.0%)	222 (82.5%)	49 (54.4%)
Codex-family agent	66 (26.0%)	47 (17.5%)	41 (45.6%)
Worktree backend	254 (100%)	269 (100%)	12 (13.3%)
Container backend	0	0	78 (86.7%)

Commit co-author trailers provide the only exact implementation-model labels:

Phase	Implementation-model evidence in commit trailers
Phase 1	Predominantly Claude Opus 4.6 and Opus 4.6 (1M context); a few early Sonnet 4.6 mentions
Phase 2	Claude Opus 4.7 (1M context), plus generic Claude (Cowork) trailers
Phase 3	Opus 4.8, Sonnet 5, Fable 5, and Opus 5, plus generic Claude (Cowork) trailers

The first later-model mentions are Opus 4.8 on June 18, Sonnet 5 on July 7, Fable 5 on July 8, and Opus 5 on July 29. These are commit-level provenance, not a one-model-per-run measurement: a squash commit can carry multiple labels, and generic Cowork trailers do not identify a model. The exact models used by Codex implementation runs cannot be recovered from the frozen state.

Who reviewed, and where

The review_caught statistics in this report measure the independent Codex review-decision gate. They do not combine every review comment from every bot.

Review-result source by implementation-reaching run	Phase 1	Phase 2	Phase 3
Local Codex review	130 (51.2%)	267 (99.3%)	86 (95.6%)
Codex in GitHub Actions	87 (34.3%)	0	0
Transition artifact with unknown source	18 (7.1%)	0	0
No durable review-result artifact	19 (7.5%)	2 (0.7%)	4 (4.4%)

GitHub-hosted Codex review artifacts span March 3 through March 16. The first explicit local-review artifact is March 17. Local review ran codex CLI against a disposable worktree and published its structured decision back to the PR; later orchestrated PRs declared local ownership so the GitHub workflow did not call the cloud reviewer again.

The exact local Codex model is not durably recorded per run. A small number of Phase 1 and Phase 2 artifacts explicitly say codex-gpt-5; most say only codex. It would therefore be too strong to label all local review as GPT-5.

The GitHub path used openai/codex-action@v1. Its model floated and was not durably recorded per review until PR #963 pinned the fallback workflow to gpt-5.5 on July 9. That pin followed the action default moving to unavailable gpt-5.6-sol and failing the gate. Because all explicit Phase 2 and Phase 3 review artifacts are local, the gpt-5.5 pin does not identify the model behind those measured local reviews.

There was also a conditional Claude reviewer in GitHub Actions using claude-sonnet-4-6, focused on spec adherence, architecture, and test coverage. Its comments are auxiliary evidence; they are not the blocking decision source counted in the reviewer catch rates above.

Models, agent-family mix, infrastructure, product maturity, and work mix changed together. The three-phase trends describe the pipeline; they cannot isolate a model effect.

Defect ledger: full history

All 1,016 PRs were reread under one semantic rubric. The 273 repair-candidate PRs represent 262 underlying engineering defects after related repair sequences were clustered and PRs that fixed independent failures were split:

• 175 strict bugs, 32 incomplete-feature defects, and 55 UX defects.
• 76 infrastructure defects and three test-system defects.
• Excluding those leaves 183 product/customer defects: 104 strict correctness bugs, 24 incomplete features, and 55 UX defects.
• Product/customer domains are 67 calculation/model, 39 product behavior, 20 data, 55 UX, and two security/privacy defects.

Examples include incorrect country tax policy, omitted capital gains in ACA MAGI, broken Google sign-in handoffs, stale solver state, contradictory MCP schemas, ignored positive savings, a Vietnam/US dual-tax crash, and local-currency amounts crossing a USD policy boundary without conversion.

Repair workload

Repair share asks what each phase worked on. It does not assign the defect to the phase that introduced it.

Fix/merge phase	Eligible product PRs	Strict product fix PRs	Broad product repair PRs
Phase 1	177	34 (19.2%)	59 (33.3%)
Phase 2	224	42 (18.8%)	77 (34.4%)
Phase 3	103	20 (19.4%)	43 (41.7%)
All phases	504	96 (19.0%)	179 (35.5%)

By this measure, the strict bug-fix burden did not go down; it stayed near 19%. Broad repair work increased in Phase 3. That can coexist with better newer code because Phase 3 fixes include defects introduced by older work, and one repair PR can address more than one independent defect.

Defect origins over time

Origin attribution uses explicit PR evidence first, then pre-fix changed-line blame, then the most recent prior product-file touch. An origin must have merged before the first fix in its repair sequence. Origins resolve for 222/262 engineering defects and 180/183 product/customer defects; attribution is still a proxy rather than proof of causality.

The raw lifetime view favors newer phases, but gives them less time to fail:

Origin phase	Eligible origins	Median observation	Strict defective origins	Broad defective origins
Phase 1	177	147 days	30 (17.0%)	46 (26.0%)
Phase 2	224	105.5 days	23 (10.3%)	42 (18.8%)
Phase 3	103	16 days	7 (6.8%)	14 (13.6%)

A fixed 30-day window makes the comparison fairer by including only origins that had 30 days to reveal a defect and counting only defects first repaired within those 30 days:

Origin phase	30-day-mature origins	Strict defective origins	Broad defective origins
Phase 1	177	24 (13.6%)	41 (23.2%)
Phase 2	224	19 (8.5%)	36 (16.1%)
Phase 3	43	3 (7.0%)	6 (14.0%)

The 14-day strict rates show the same direction: 12.4%, 7.1%, and 5.0% across 177, 224, and 60 mature origins. The honest conclusion is that the observed defect-introduction rate went down, including under equal exposure, while the amount of repair work did not. Phase 3’s mature samples are still small, and model, workflow, infrastructure, work mix, and product maturity changed together.

Origin change size

Origin PR churn	Origin PRs	Strict-defective PRs	Defective rate	Strict defects/K changed lines
0-49	22	0	0.0%	0.000
50-199	76	1	1.3%	0.109
200-499	118	9	7.6%	0.213
500-999	119	17	14.3%	0.246
1,000-1,999	118	17	14.4%	0.182
2,000+	51	16	31.4%	0.198

The chance that a PR is later implicated rises sharply with change size: 31.4% for 2,000+ lines versus 9.7% below 2,000. Defects per changed KLOC do not rise in the same way because the denominator scales with PR size. The actionable result is about review and failure surface per change: split the largest specs and PRs.

Origin work mode

Origin provenance	Eligible origins	Broad-defective origins	Strict-defective origins	Broad defects/K changed	Strict defects/K changed
spec implement	279	59 (21.1%)	32 (11.5%)	0.265	0.143
spec task	85	10 (11.8%)	5 (5.9%)	0.318	0.139
Other code PR	140	33 (23.6%)	23 (16.4%)	0.681	0.443

Tasks have the lowest defective-origin rate, but they are much smaller changes. Their strict defect density per changed KLOC is almost identical to full specs (0.139 versus 0.143). The higher rate for other code PRs is also confounded: this bucket contains substantial early work whose run provenance was not durably mapped, plus ad hoc corrective work. It does not mean manually authored code.

Density

Two retrospective ratios are useful only as scale checks:

• 104 / 187.935 current non-test KSLOC = 0.55 known repaired strict product defects per KLOC.
• 183 / 187.935 = 0.97 known repaired broad product/customer defects per KLOC.

The earlier 0.74 figure combined 110 defects from the old cumulative rubric with 29 from the later rubric. It is retired. The consistent full-history ledger produces 0.55 strict and 0.97 broad, which explains why earlier summaries appeared close to one defect/KLOC depending on the definition. Neither is a remaining-defect density: the numerator is historical repaired defects and the denominator is the current tree after deletion and replacement.

Comparison with traditional development

There is no clean manual-team control group. Commits/day, defects/KLOC, corrective-work share, and review catches all measure different constructs. The useful comparison is a set of reference points, not a single league table.

Corrective-work share

The Corrective Commit Probability study examined 7,557 active GitHub projects. Its valid-project distribution had a 20% median corrective-commit probability, 6% at the high-quality 90th percentile, and 39% at the low-quality 10th percentile.

Retirement Lab’s full-history 19.0% strict product-fix PR share is close to that 20% median. The comparison is approximate: this ledger uses semantic PR descriptions and a product-only denominator, while CCP uses a linguistic commit classifier. The 35.5% broad share includes incomplete-feature and UX follow-ups; it measures additional rework and should not be compared with the study’s bug-fix rate.

A manually labeled study of 3,498 bug-fixing commits found that only 25.1% of changed lines in those commits were actual bug-fix lines; tests were 39.6% and documentation 14.0%. That is why this report classifies PR intent and defects instead of treating every line in a fix PR as a defect.

Defect density

A scoping study of 109 projects reported a median 4.3 and mean 7.47 post-release defects/KLOC, with substantial variation between projects. Retirement Lab’s full-history PR ledger contains 0.55 known repaired strict defects per current KLOC, or 0.97 when incomplete-feature and UX defects are included. The definitions and observation processes are not identical, so these figures do not establish that Retirement Lab is four to eight times better. They are nevertheless encouraging: all known repair work flows through the repository, the broader measure includes more than conventional correctness bugs, and the observed density remains well below the study’s median.

Retirement Lab is young, so additional defects will be discovered and its recent work is especially right-censored. At its current size, however, the broad ledger would have to grow from 183 to roughly 808 defects to reach the study median. The current-tree denominator also does not perfectly match the historical defect numerator because code has been added, replaced, and deleted. The reasonable conclusion is not superiority, but that the quality observed so far is numerically favorable and provides legitimate cause for cautious optimism.

Change size and reviewability

Google’s small-change guidance says 100 lines is usually reasonable and 1,000 lines is usually too large; smaller changes are reviewed more thoroughly and are less likely to introduce bugs.

The repository’s own outcome data agrees with the direction of that guidance: full-history origin PRs with 2,000+ changed lines had a 31.4% strict-defective rate, versus 9.7% below 2,000. Phase 3 also has the largest share of 2,000+ commits at 13.1%. This is a stronger and more honest comparison than claiming an abstract code-quality score.

Testing practice

The same Google guidance says tests are expected for changes that add or alter logic. Retirement Lab improved from 80.4% product-change accompaniment in Phase 1 to more than 91% in Phases 2 and 3. Full-spec PRs were almost always accompanied by tests; tasks were the main Phase 1/2 gap. Because no coverage gate is configured, the repository cannot be compared on branch or statement coverage.

Throughput and shipped value

The September 2026 revision of an NBER study of more than 500,000 GitHub developers found cumulative effects of autonomous agents of roughly +240% on commits, but only +80% on projects and +30% on releases. That is a useful warning against turning Retirement Lab’s commits/day into a manual-team equivalence claim. The live product is the best evidence that the work shipped; release, adoption, reliability, and business outcomes are better follow-on measures than commit volume.

External AI evidence also depends heavily on workflow. METR’s early-2025 randomized trial found 16 experienced open-source developers took 19% longer with then-current AI tools across 246 issues. That setting used familiar developers doing two-hour issues with interactive tools; Retirement Lab uses a non-coding operator and long autonomous loops with independent machine QA. The results are not contradictory, but they are not directly comparable.

DORA is intentionally absent. The repository snapshot does not provide a trustworthy deployment-frequency, lead-time, failed-deployment, and restoration ledger, and those measures would not answer the narrower question of defect quality anyway.

What the data supports

1. A non-coding operator can sustain a real software product. The current tree has 187,935 non-test programming-language lines and 124,199 test lines, and the live product is independently inspectable.
2. Most orchestrated implementations complete without a keyboard return. The primary mapped-run estimate improves from 81.9% to 90.7% across the three phases.
3. The workflow is not unreviewed generation. Review or tests rejected 65%-80% of implementations at least once, including concrete correctness, concurrency, entitlement, and data-integrity errors.
4. Quality did not obviously collapse as the codebase grew. Test accompaniment improved, test volume grew, aggregate complexity density fell, and the former worst hotspot was decomposed.
5. Observed defect introduction improved across the three phases. The 30-day strict defective-origin rate fell from 13.6% to 8.5% to 7.0%, though Phase 3 has only 43 mature origins.
6. Agent-written software still has ordinary software risks. The full-history audit found 104 repaired strict product defects; broad repair work consumed 35.5% of eligible product PRs.
7. The workflow’s biggest measured risk is oversized changes. Defect concentration above 2,000 changed lines is too large to ignore, and those changes became more common in Phase 3.

What it does not support

• That manual coding is dead for every developer, codebase, or organization.
• That Retirement Lab has better quality than a strong manual team.
• That 65%-80% of potential production defects were prevented by the reviewer/tests.
• That newer models caused the phase improvements; model, infrastructure, work mix, and project age changed together.
• That 3.0 or 13.6 commits/day equals a particular number of human engineers.
• That test SLOC or declaration count is coverage.
• That Phase 3’s current 30-day result is a settled long-run rate; only 43 origins are mature enough for it.

Next measurements worth adding

• Statement and branch coverage, frozen at every release.
• Deployment and rollback ledger tied to PRs.
• Production exceptions, incidents, and customer-reported defects with first-seen and fixed dates.
• Spec-start to merge lead time and agent compute cost per merged PR.
• Refresh the 30-day cohort and add 60/90-day horizons as Phase 3 matures.
• PR-size policy and before/after defect rate once 2,000+ line changes are split.
• Release/adoption outcomes so throughput can be discussed as shipped value rather than commit activity.

Artifact map

• data/development-phases-consolidated-2026-08-14.json: frozen three-phase throughput, run, QA, intervention, mode, stall, and endpoint code metrics.
• data/development-phase-endpoints-2026-08-14.json: exact code-tree and test-declaration inputs at PRs #520, #911, and #1101.
• scripts/analyze_development_phases.py: reproducible phase aggregation from the frozen repository, PR, run, and endpoint data.
• data/run-metrics-consolidated-2026-08-14.json: intervention, stalls, reviewer, tests, modes, agents, and per-run rows.
• data/repository-consolidated-2026-08-14.json: commit/day, churn, test accompaniment, and per-commit rows.
• data/pr-metrics-full-history-2026-08-14.json: PR size, file categories, run mapping, and one-rubric semantic class for all 1,016 PRs.
• data/defect-items-full-history-2026-08-14.json: 262 reconciled underlying defects represented by 273 repair PRs.
• data/defect-ledger-full-history-2026-08-14.json: full-history origins, phase rates, fixed observation windows, size rates, and work-mode rates.
• data/repair-rates-full-history-2026-08-14.json: strict and broad repair workload by phase and PR provenance.
• data/review-test-escape-full-history-2026-08-14.json: full-history non-causal catch/escape correlation.
• data/code-metrics-consolidated-2026-08-14.json: same-method SLOC and complexity at PRs #797 and #1101.
• data/code-hotspots-consolidated-2026-08-14.json: file-size and complexity hotspots.
• data/test-inventory-consolidated-2026-08-14.json: static test declarations at PRs #797 and #1101.
• data/model-evidence-consolidated-2026-08-14.json: implementation families, trailer model labels, review location by phase, and reviewer-model evidence.
• methodology/defect-classification-rubric.md: semantic classification rules.

Important implementation audit corrections retained during consolidation:

• Origins in a multi-PR repair sequence must predate the first fix PR; an earlier PR in the same repair sequence cannot be called the introducing origin.
• Older GitHub Actions job IDs are not parsed as PR numbers. Merged-run metrics join through the explicit run ID in the PR ledger or an actual /pull/<number> URL.