The final recall number is 17 out of 18. Three AI agents reviewed 8 seeded pull requests containing 18 planted bugs, and they caught 17. 100% precision with zero false positives and 100% agreement with my own human review. The full eval run cost $0.54 and the one miss is reported honestly in the README instead of buried.

I've built PR and change request automation professionally, so this isn't my first pass at the problem. AgentReview is a public, from-scratch take on something I know well, built to measure what multi-agent code review actually costs, catches, and misses. The live demo is up. The code is on GitHub. Let me walk through the numbers.

The Architecture in 30 Seconds

An orchestrator takes a PR diff and fans it out to three specialized agents concurrently, focusing individually on code quality, security, and documentation. Each agent calls real tools through MCP (Model Context Protocol). I built a static-analysis MCP server using the official MCP C# SDK that exposes Roslyn analyzers and Semgrep. The agents also use GitHub's hosted MCP server to pull surrounding file context so they're not reviewing diffs in a vacuum.

This was built with plain C# orchestration instead of using LangChain, Semantic Kernel, or an agent framework. This just uses concurrent fan-out with tasks and keyed dependency injection. The thesis is the same one I proved with DocQuery for RAG. Agentic tooling is overwhelmingly Python, and the same patterns work just fine in the Microsoft stack. AgentReview is DocQuery's sibling solving a different problem with the same C# orchestration argument.

The Numbers From Real Runs

The evals are part of the deliverable. Eight seeded PRs with 18 planted bugs across them. Here's what came back:

One real run from the live demo in the browser resulted in 10 ranked findings, 2 duplicates merged across agents, 21.5 seconds, $0.081. Those cost numbers aren't estimates since everything is traced with OpenTelemetry, with per-agent spans, tool calls, and token counts per LLM call. The cost figures come from span data.

I'm not going to pretend 17/18 is 18/18. The miss is a real gap, and it's documented. If you're evaluating this kind of system, the honest miss tells you more about reliability than a perfect score on a small benchmark ever could.

The Surprise: Why Synthesis Needs Provenance, Not String Matching

The thing I didn't expect was that the quality agent's LLM restated Semgrep's SQL injection finding in its own words. Two agents with the same vulnerability, but with completely different phrasing. If you're deduplicating findings with string matching or even embedding similarity, you'll either miss the duplicate or merge things that shouldn't be merged.

So, synthesis in AgentReview works differently. An LLM arbiter clusters findings that restate the same issue, but it never rewrites them. Then, deterministic rules pick the survivor with a hierarchy of tool-backed beating LLM-only, then higher severity wins, then security over quality over docs. Every finding keeps exact structure using roslyn, semgrep, quality-llm, security-llm, or docs-llm.

That chain matters because when the quality agent's LLM says "this SQL query is concatenating user input" and Semgrep's rule says "SQL injection via string concatenation," the arbiter clusters them and the Semgrep-backed finding survives. You get the tool's precision with the LLM's natural language, and you know exactly where the finding came from. The 2 duplicates merged in that live demo run were both this pattern with agents independently flagging the same issue through different lenses.

Cost Guardrails Are Code, Not Policy

When I've seen teams deploy LLM-powered automation internally, cost control usually lives in a wiki page or a Slack reminder. That doesn't scale, so in AgentReview I built cost guardrails into the code.

Per-agent token caps and a per-review budget are enforced before execution. A run whose worst-case cost exceeds the budget refuses to start. It doesn't run and then apologize because it won't begin in the first place.

For the public demo, I locked it down further. The demo reviews only six committed sample diffs and a free-form input returns 403. There's a per-IP rate limit and a global daily cap of 25 reviews, which works out to about $1.75 per day worst case. I can leave it running without checking my Azure bill every morning.

// Simplified: the actual guard checks worst-case token estimates
// against the budget before any agent starts work
if (estimatedWorstCaseCost > reviewBudget)
{
    logger.LogWarning("Review rejected: estimated cost {Cost} exceeds budget {Budget}",
        estimatedWorstCaseCost, reviewBudget);
    return ReviewResult.BudgetExceeded(estimatedWorstCaseCost, reviewBudget);
}

The OpenTelemetry traces feed directly into this. Token counts per LLM call are the source of truth for cost accounting. When I say $0.068 per review, that's summed from span-level token counts, not napkin math.

Why C# and Not the Python Stack

I keep getting this question, and the answer is the same one I gave when I built DocQuery. If your production stack is .NET, building your AI tooling in Python means maintaining a second world for no architectural reason. Tasks, keyed DI, and IHostedService give you everything you need for agent orchestration. MCP has a first-party C# SDK, and Roslyn analyzers are already .NET. The integration surface is smaller when you stay in one stack.

AgentReview doesn't use any agent framework because it doesn't need one. The orchestrator fans out tasks to each agent, and they all have a tool-calling loop. Synthesis runs after all agents complete. Adding a framework would give me abstractions over things I can already see and control.

What This Actually Tells You

If you're thinking about multi-agent code review, or multi-agent anything, here's what I'd focus on:

Measure from the start. OpenTelemetry isn't something you add later. If I hadn't had per-agent spans from day one, I wouldn't have caught that the quality agent was restating tool findings instead of adding new signal. Traces are how you debug agent behavior, not just performance.

History of ownership is non-negotiable. The moment two agents can see the same code, they'll produce overlapping findings. You need to know which tool or model produced each finding, or your deduplication logic will either drop real issues or surface noise.

Budget enforcement belongs in code. If a run can exceed your cost threshold and you find out after the fact, your guardrail isn't a guardrail. Check before you start.

Report the miss. 17/18 is a more useful number than 18/18 on a benchmark you control. The miss tells you where your system's blind spots are. I'd rather ship honest recall than perfect-looking evals.

AgentReview is live at agentreview.vondraysanford.com. Try the sample diffs, look at the traces, and check the cost. The code's at github.com/vondraysanford/AgentReview. The one bug it missed is in the README.