Retirement Lab evidence

A reviewed change that still shipped a bug

The origin spec, intervention, eight rejected reviews, later strict product defect, two-line fix, and 165-line regression test for Retirement Lab’s Timeline editor.

PR #834 introduced a dedicated Timeline page. It was a large change: 3,365 additions and 305 deletions across 12 files, including a new 1,885-line component. The pipeline rejected it eight times, reported 15 findings, required an explicit operator response, and ultimately approved it. A strict product bug was nevertheless discovered and repaired 35 days later in PR #913.

What the gates caught

The independent reviewer found 11 P1 and 4 P2 issues before merge, including dropped Roth conversions, incomplete event editors, stale relocation adjustments, incorrect peak totals, missing mobile behavior, and several save/edit regressions.

One finding was close to the later defect: attempt 8 reported that editing a benefit’s claim age did not update its start year. That issue was addressed, but the input still used defaultValue and copied the visible value into React state only on blur.

What escaped

If a user changed claim age and immediately clicked Save, the input displayed the new age while the save payload still contained the previous claim_age and stale begin_year. PR #913 changed the field from uncontrolled defaultValue/onBlur behavior to controlled value/onChange behavior.

The production fix changed two product lines and added a 165-line jsdom regression test that edits an “Other government pension” claim age and saves without requiring blur. The repaired lines traced back to code introduced in PR #834.

Intervention and run record

FieldValue
Origin size3,670 changed lines
Implementation and reviewClaude implementer; separate Codex reviewer
Iterations13 implementation retries and 8 rejected reviews
Review findings15
Human interventionOne explicit operator response; retry limit raised
Repair lag35 days

The run eventually stopped on a recurring merge conflict and asked the operator to intervene. The response resolved the conflict and one outstanding review finding.

Source material

What it demonstrates

The review loop materially improved the PR: the preserved findings identify concrete product defects that did not ship in that form. The later bug also shows why the reviewer catch rate is not a prevented-defect rate. A reviewer can find a nearby failure mode and still miss a distinct event-ordering problem in the same control.

This is evidence against both extreme readings. The agents were not blindly merging first drafts, and the independent gates were not sufficient to guarantee correctness. Large change size remains the clearest measured risk in the aggregate ledger: 31.4% of eligible 2,000-plus-line origin PRs later revealed a strict product defect, versus 9.7% of smaller origins.