Skip to content

Ask the review agent to check behaviour, not read the diff - #35

Merged
savvasioakeim merged 1 commit into
mainfrom
feat/review-agent-method
Oct 31, 2026
Merged

savvasioakeim merged 1 commit into
mainfrom
feat/review-agent-method

Conversation

@savvasioakeim

Copy link
Copy Markdown
Owner

The review agent's entire instruction was "give a concise review: what's good and any real problems". That is why it kept concluding the code looked fine. This gives it a method.

What carries the weight

Read past the diff. In risky spots it reads the whole function and every caller, not the changed lines. The bugs in this codebase live at call sites the diff never shows.

An evidence bar. Every finding must arrive with four things: a location, a concrete trigger (input, sequence or interleaving), the real consequence for the user, and why it is not already prevented elsewhere. Without them it must say it suspects something rather than presenting it as confirmed. Style preferences and hypotheticals with no reachable path are ruled out explicitly. This is the part that keeps the output short enough to be worth reading.

The risk list is ours, not a generic one

Each entry is a class of bug this app has actually shipped:

  • state read outside a render (refs assigned during render, module stores, handler closures)
  • Tauri listeners that outlive their window, or register twice
  • the main and QA webviews having separate capability files, so a new plugin command fails only in the window nobody tested
  • persisted settings shape on a machine that upgraded from an older version
  • process and session lifecycle: PTYs, spawned agents, a turn in flight at cancellation
  • Windows paths
  • backends that degrade silently instead of failing

The last one comes from the screen capture returning black pixels. The same day, it caught us again somewhere else: a dictation onerror that swallowed its code had been hiding a broken feature for an unknown length of time.

Concurrency as a procedure

Write the invariant, split reads from writes, interleave two executions at every await, don't assume an earlier check survives an await, and look at what a failure between two writes leaves behind.

Safety

The ```octo-review-verdict block is byte for byte unchanged. parseVerdict matches it by regex and aggregateReviews gates QA on the result, so editing it would have broken the gate quietly.

Provenance

Adapted from a human reviewer's own description of how she reviews. Her domain checklist (per-tenant permissions, money, timezones, migrations, lease expiry, BE/FE deploy ordering) was dropped, since none of it exists in this app, and replaced with the list above. The method — behaviour over diff, tests first, the evidence bar, the concurrency procedure, and regression tests that must fail before the fix — transferred intact.

Verification

npx tsc --noEmit clean. The prompt is data, so there is nothing to execute here; the real test is the next review this agent produces.

🤖 Generated with Claude Code

The whole instruction was "give a concise review: what's good and any real
problems", which is why the reviewer kept concluding the code looked fine.
This gives it a method.

The two parts carrying most of the weight: in risky spots it reads the
whole function and every caller rather than the changed lines, because the
bugs here live at call sites the diff never shows; and every finding has to
arrive with a location, a concrete trigger, the real consequence, and why
it is not already prevented elsewhere. The second is what keeps the output
from filling with style notes and unreachable hypotheticals, and it is
explicitly allowed to say "I suspect" instead of overclaiming.

The risk list is this codebase's own, not a generic one. Each entry is a
class of bug we have actually shipped: state read outside a render,
listeners that outlive their window, the two webviews with separate
capability files, persisted shape on an upgraded machine, process and
session lifecycle, Windows paths, and backends that degrade silently
instead of failing. That last one came from the capture returning black
pixels, and the same day it caught us again in a different place.

The concurrency section is a procedure rather than advice: write the
invariant, split reads from writes, interleave two executions at every
await, and look at what is left behind by a failure between two writes.

The octo-review-verdict block is byte for byte unchanged. parseVerdict
matches it by regex and aggregateReviews gates QA on the result, so editing
it would have broken the gate quietly.

Method adapted from a human reviewer's own description of how she works.
Her domain checklist (per-tenant permissions, money, migrations, deploy
ordering) was dropped — none of it exists in this app — and replaced with
the list above.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@savvasioakeim
savvasioakeim merged commit 099219c into main Oct 31, 2026
3 checks passed
@savvasioakeim
savvasioakeim deleted the feat/review-agent-method branch October 31, 2026 18:28
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 31, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant