Claims are not evidence: verifying what an AI agent tells you it fixed

Reviewing a change used to mean reading the code. You opened the diff, you understood the intent, you looked for the mistake. The work was comprehension.

That is shifting. When an agent writes the change, writes the description, writes the test, and then writes a summary saying the issue is resolved, the reviewer arrives as the first auditor of a set of assertions rather than the first reader of the code. Comprehension still matters. It has stopped being the scarce skill. The scarce skill now is deciding which assertions have been checked and which have only been stated.

We use these tools daily and they are a large net gain, so take the following as a narrow point rather than a complaint. The artifacts that used to be weak evidence of work have become very cheap to produce, and our review habits have not caught up.

Everything below comes from our own review practice on production codebases.

Seven of eight replies were false

We ran a reconciliation sweep across eight merge requests. In each one, a review finding had been answered with some version of "fixed in a follow-up commit". The sweep checked whether the named file had actually changed.

Seven of the eight replies were false. The files had been byte-identical for three weeks.

Nobody in that thread was lying. A reply like that is usually written in good faith by somebody who intended to make the fix, believed a later commit had covered it, or lost track across a long-running branch. The failure was structural. A reply closed a finding, and no step existed between the reply and the merge that checked whether the reply was true.

Once you have seen this, the phrase "fixed in a follow-up" reads differently. It is a claim about a state of the repository, and the repository can be asked directly.

Verify with blob identity, not a diff summary

The technique we settled on is small. Take the file path, resolve the stored object hash for that path at the commit where the finding was raised, resolve it again at the tip of the branch, and compare the two hashes.

If the hashes match, the file has not changed at all. Not reformatted, not touched, not partially addressed. There is no room to argue about whether a rename or a whitespace pass counts, because content addressing does not care about intent. A diff summary invites argument. Blob identity does not.

This is faster than reading the diff, it can be scripted across every open finding on a branch, and it produces a result that a person can verify independently in ten seconds. Those three properties matter more than sophistication. A verification step that takes effort will be skipped under deadline pressure, and a verification step that can be disputed will be disputed.

Two caveats keep it honest. Identical hashes prove nothing changed, which is the strong direction. Different hashes only prove something changed, so the second question, whether the change addresses the finding, still needs a human. The technique removes an entire category of false closure. It does not remove review.

Descriptions drift away from the code

On one batch of nine changes, we checked every description against what the change actually did. Three were wrong. One of them described the wrong set of affected platforms, which is the kind of error that survives review easily because nobody rereads the description after the first pass.

Descriptions drift for an ordinary reason. They are written early, when the change is a plan. The code then moves, scope gets trimmed, a second approach replaces the first, and the prose at the top of the request keeps describing the original plan. An agent writing the description from an early draft of the work produces the same drift faster.

Treat the description as an artifact under review, with the same standing as a line of code. Read it last, against the final diff, and ask a specific question: if someone six months from now reads only this paragraph, will they believe something false?

Green CI is not evidence

A passing pipeline feels like proof. It often is not.

We found a test that passed identically before and after the fix it was supposed to cover. It exercised a path the change did not touch. Its green result carried no information about the defect at all, but it appeared in the request as though it did.

We also found tests that never executed. An unrelated stage failed earlier in the pipeline, later stages were skipped, and the overall status still rendered in a way that invited a reviewer to move on. Nobody had checked whether the test in question ran.

Two questions catch most of this. Did this test fail before the fix? Did it actually execute in this run? Until a test has been shown failing against the broken code, all you know is that it passes, which is a much weaker statement than the one the pipeline appears to be making.

Duplicate review is now the default, not an edge case

Everyone is running similar agents against similar code with similar prompts. The consequence shows up quickly in shared repositories.

We watched two reviewers who had no visibility into each other produce the same finding on the same lines. Not a similar concern in the same area. The same finding, in nearly the same words.

The cost is real. Authors receive the same objection twice and respond to it twice, or respond once and leave the second reviewer's comment open, which then looks like an unresolved blocker. Review queues fill with duplicates, and duplicates train people to skim.

The practical response is to read what is already on the thread before adding anything, and to say plainly when you are confirming an existing finding rather than raising a new one. That sounds obvious. It stopped being automatic the moment generating a thorough-looking finding became nearly free.

Attack your own findings before you deliver them

We added an adversarial second pass to our own review process. Before a finding is published, somebody tries to refute it: read the surrounding code, look for the guard that makes the concern impossible, check whether the framework already handles it.

The pass earns its keep. One of our own blocking findings was refuted outright during it. We retracted it and published the retraction rather than quietly dropping it, because a retraction that other people can see is the thing that makes the next finding credible.

An agent will produce findings with uniform confidence regardless of whether they are correct. That uniformity is the problem. If the presentation does not distinguish a certain finding from a speculative one, the distinguishing has to happen before delivery, and it has to be somebody's explicit job.

The governance point, for people who do not read diffs

Strip out the engineering detail and a general principle remains. An agent that reports success is reporting a claim. So is a status update in a meeting, a green check on a dashboard, and a note that says the issue was handled last week.

The useful question is always the same. What artifact would look different if this claim were false? Then go and look at that artifact instead of at the claim.

Sometimes the artifact is a file hash. Sometimes it is a row in a table, a log line, a record whose timestamp would have to have moved. The discipline generalizes well beyond code review, and it costs far less than the alternative, which is finding out at the point where the claim finally meets reality.

Three changes are worth making this quarter. Require that every closed review finding is verified against the repository rather than against a reply. Require that a regression test is shown failing before it is accepted as proof. And when a tool tells you something is done, decide in advance which artifact you would inspect to prove it wrong, then inspect it.