# AgentReview Phase 5: Measuring My Own Agents, and Why Precision Needs a Human

Published: 2026-08-09 · Tag: AI
Canonical: https://vondraysanford.com/writing/2026-08-08-agentreview-phase-5-measuring-my-own-agents-and-why-precisio.html
Author: Vondray Sanford (https://vondraysanford.com)

> Recall is easy to measure when you plant the bugs yourself. Precision is the metric that fights back. It took 45 manual judgments, a scoring worksheet, and a budget guard wired to every token cap before I trusted these numbers.

Shipping agents is one thing, but measuring them honestly is something else. I've been building AgentReview, a multi-agent PR review pipeline, for a while now, and Phase 5 was dedicated entirely to evaluating what was already built. This phase had no new features and no refactors, just a rigorous look at whether the agents catch problems and whether what they flag is worth flagging. The numbers came out better than I expected, but getting there required a process I wasn't willing to skip.

## Recall Is Cheap When You Plant the Bugs Yourself

The cleanest way to measure recall is to control the ground truth. I created 8 seeded diffs, synthetic pull requests where I deliberately introduced 18 labeled issues. Off-by-one errors, unchecked nulls, logic inversions, the kind of bugs that actually matter in a review. Then I ran the pipeline and checked how many it caught.

Under strict exact-line matching, the agents found 17 of 18. That's a strong number, and the total cost for all 8 seeded reviews came in at roughly fifty cents, but seeded recall is a lower bound on real-world difficulty. I designed those bugs, which means I know exactly what a good detection looks like. The pipeline doesn't get that on an actual PR.

The one miss is worth more to me than the 17 hits. I know what kind of issue it was, I know which agent skipped it, and that gives me a concrete improvement target for the next phase.

## Precision Is the Hard Metric, So I Built Tooling to Judge It Fairly

An unplanted finding is not automatically wrong. If an agent flags a real problem that I didn't seed, that's a true positive, not a false alarm, but you can't automate that judgment. You have to actually read the finding and decide whether it's legitimate. Every unmatched finding had to go through a human review pass.

Across the 8 seeded reviews, the pipeline surfaced 45 unmatched findings. I built a human worksheet workflow and went through all 45. The key decision I made early was that I wasn't going to judge findings from raw JSON and diff fragments. My tooling generated a review guide that reconstructed the relevant code context and paired it with each pending finding, basically the same view you'd have if these were comments on a real PR. Same information, same format, same standard I'd apply to a colleague's review.

The result of the manual review was 100% precision. Every unmatched finding I judged was either a legitimate catch or a duplicate of a seeded issue caught by a different mechanism. I also re-evaluated my own judgments with fresh eyes and didn't flip a single call. That matters because evaluation reproducibility is its own metric. If your human judgments aren't stable, your evaluation numbers aren't either.

## The Deduplication Insight I Didn't Expect

There was one surprising finding buried in the results. My synthesis layer runs a dedupe step so when multiple agents catch the same issue, it picks the best finding and discards the rest. That's the right behavior because you don't want three agents all commenting on the same null dereference in a real PR. But, the dedupe re-attributes the catch to the surviving agent.

On paper, this costs two label matches. If Agent A and Agent B both catch a seeded issue and the synthesizer keeps Agent B's finding, Agent A shows zero credit for that catch in the raw match data. That could make an agent look weaker than it is, or make the recall math misleading if you're counting at the agent level instead of the pipeline level. I'd rather measure pipeline recall, what the user sees, but it's worth logging agent-level catches before deduplication if you want to tune individual agents later, which I've flagged that for Phase 6.

## Observability and Budget Guards Underneath All of It

The evaluation infrastructure is more than just a spreadsheet. I've got OpenTelemetry spans on every agent, every tool call, and every LLM call. Those spans aggregate into a per-review cost line so I can see exactly what each review cost, broken down by component. That's how I got to the fifty-cent figure with confidence, which is the sum of the traced token usage across all 8 reviews.

I also built a budget guard that runs before any review starts. It computes a worst-case cost estimate from the configured token caps and refuses to start the review if it can't afford to complete it. This sounds conservative, but it's the right call. A pipeline that blows past your budget halfway through and returns a partial result is worse than one that declines upfront. You get a clear error, you adjust your caps or your input, and then you try again. No surprise bills and no silent truncation.

The full numbers, recall rate, precision, cost per review, agent-level match breakdown, are in the project's README. I'd rather point you there than paste a table here that goes stale the next time I update the pipeline. The methodology for how I generated the seeded diffs, built the worksheet, and computed the final metrics is documented alongside the results so you can reproduce it or adapt it for your own agent eval.

## If You're About to Evaluate Your Own Pipeline

DO NOT skip the human worksheet step, even if it's tedious (I even contemplated having Claude do this for me lol). Precision without manual judgment of the unmatched findings is just recall with extra steps. Build your own judgment tooling so you're evaluating in the same context the findings are meant to live in. Judging from raw diffs and JSON is how you end up with inconsistent calls you can't defend.

Also, watch your deduplication logic carefully. It's doing work, and that work has downstream effects on how you read your own metrics. Know whether you're measuring pipeline output or per-agent output, and be explicit about which one you care about.

**Wire up cost observability before you need it, not after.** OpenTelemetry span coverage on LLM calls is the only way to have an honest conversation about whether the pipeline is economical enough to run at scale. Fifty cents for 8 reviews is a fine number, but I want to know if it stays fine when the PRs get larger and the agents get more capable.

Phase 6 is already taking shape. The miss from the seeded recall set is first on the agenda.