Repository navigation
Ask the review agent to check behaviour, not read the diff - #35
Merged
Merged
Conversation
The whole instruction was "give a concise review: what's good and any real problems", which is why the reviewer kept concluding the code looked fine. This gives it a method. The two parts carrying most of the weight: in risky spots it reads the whole function and every caller rather than the changed lines, because the bugs here live at call sites the diff never shows; and every finding has to arrive with a location, a concrete trigger, the real consequence, and why it is not already prevented elsewhere. The second is what keeps the output from filling with style notes and unreachable hypotheticals, and it is explicitly allowed to say "I suspect" instead of overclaiming. The risk list is this codebase's own, not a generic one. Each entry is a class of bug we have actually shipped: state read outside a render, listeners that outlive their window, the two webviews with separate capability files, persisted shape on an upgraded machine, process and session lifecycle, Windows paths, and backends that degrade silently instead of failing. That last one came from the capture returning black pixels, and the same day it caught us again in a different place. The concurrency section is a procedure rather than advice: write the invariant, split reads from writes, interleave two executions at every await, and look at what is left behind by a failure between two writes. The octo-review-verdict block is byte for byte unchanged. parseVerdict matches it by regex and aggregateReviews gates QA on the result, so editing it would have broken the gate quietly. Method adapted from a human reviewer's own description of how she works. Her domain checklist (per-tenant permissions, money, migrations, deploy ordering) was dropped — none of it exists in this app — and replaced with the list above. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The review agent's entire instruction was "give a concise review: what's good and any real problems". That is why it kept concluding the code looked fine. This gives it a method.
What carries the weight
Read past the diff. In risky spots it reads the whole function and every caller, not the changed lines. The bugs in this codebase live at call sites the diff never shows.
An evidence bar. Every finding must arrive with four things: a location, a concrete trigger (input, sequence or interleaving), the real consequence for the user, and why it is not already prevented elsewhere. Without them it must say it suspects something rather than presenting it as confirmed. Style preferences and hypotheticals with no reachable path are ruled out explicitly. This is the part that keeps the output short enough to be worth reading.
The risk list is ours, not a generic one
Each entry is a class of bug this app has actually shipped:
The last one comes from the screen capture returning black pixels. The same day, it caught us again somewhere else: a dictation
onerrorthat swallowed its code had been hiding a broken feature for an unknown length of time.Concurrency as a procedure
Write the invariant, split reads from writes, interleave two executions at every
await, don't assume an earlier check survives anawait, and look at what a failure between two writes leaves behind.Safety
The
```octo-review-verdictblock is byte for byte unchanged.parseVerdictmatches it by regex andaggregateReviewsgates QA on the result, so editing it would have broken the gate quietly.Provenance
Adapted from a human reviewer's own description of how she reviews. Her domain checklist (per-tenant permissions, money, timezones, migrations, lease expiry, BE/FE deploy ordering) was dropped, since none of it exists in this app, and replaced with the list above. The method — behaviour over diff, tests first, the evidence bar, the concurrency procedure, and regression tests that must fail before the fix — transferred intact.
Verification
npx tsc --noEmitclean. The prompt is data, so there is nothing to execute here; the real test is the next review this agent produces.🤖 Generated with Claude Code