The fan-out is the part that feels satisfying. You drop Task.WhenAll in, watch three agents fire simultaneously, and the wall clock time collapses. What used to run sequentially in 31 seconds of agent work now lands in 14 seconds of real time. MCP multiplexing over a single stdio server with three agents sharing tools and no drama or issues occuring.

Then the results come back and you realize the hard part hasn't started yet.

The Merge Problem No One Talks About

When you run three independent agents over the same diff, they don't coordinate. Each one is doing its own analysis in its own context window, and they'll independently notice a lot of the same things, just worded differently. Agent A flags a "hardcoded connection string." Agent B flags "hardcoded credentials in source." They're both describing the same problem, but there's no string similarity threshold that reliably catches that pairing without also collapsing findings that really are different.

That's one failure mode, but the other is the opposite. They find the same line with two distinct issues. A line might have both a null reference risk and a logging side-effect that shouldn't be there. If you deduplicate naively by line number, you keep one and silently drop the other. The output looks cleaner, but you've hidden a relevant issue. Neither text rules nor line-based deduplication are sharp enough tools for this problem.

I did some research into how other multi-agent pipelines handle this and the honest answer is most of them just pick one agent's output as canonical or do a dumb union and call it done. Neither option worked for AgentReview, where the whole point is that the synthesized review is more trustworthy than any individual agent's pass.

The Hybrid: LLM for Clustering, Determinism for Survivors

My answer is a two-stage synthesis pipeline. Stage one is an LLM arbiter, but a constrained one. It doesn't get to rewrite findings, summarize them, or delete anything. Its only job is to look at the full set of finding IDs and return groups of IDs that it believes describe the same underlying issue. The output is a JSON structure like this:

[
  { "cluster_id": "c1", "member_ids": ["a-003", "b-007"] },
  { "cluster_id": "c2", "member_ids": ["a-011"] },
  { "cluster_id": "c3", "member_ids": ["b-002", "c-005", "a-009"] }
]

The LLM is good at this. It can read "hardcoded connection string" and "hardcoded credentials" in context and recognize they're the same thing. But, because it's only clustering IDs and never touching the finding content, there's no hallucination risk on the output that ships to the developer.

Stage two is fully deterministic. For each cluster, we pick one survivor using a stated priority chain with the following rules. First, tool-produced findings beat LLM-produced findings, then higher severity wins, then agent priority (Security agent outranks Style agent, for example). There are no judgment calls made at runtime. If you want to know why finding b-007 survived over a-003, the rules tell you exactly why.

The reason tool beats LLM in that priority chain is intentional. When an agent calls an actual static analysis tool through MCP and gets back a structured result, that's a harder signal than a language model noticing a pattern in the diff text. We want the output to reflect that difference in confidence.

Provenance Survives the Merge

One thing I was firm about from the start is that synthesis can't be a black box. Every finding in the final output still carries its full provenance. The output lists which agent produced it, whether it came from a tool call or from the agent's own LLM analysis, and which other findings it absorbed when its cluster was resolved. If a developer wants to understand why they're only seeing one finding for a particular line, they can look at the provenance and see that two other agents agreed with the survivor's assessment.

This is what makes the system debuggable. When a finding looks wrong, you can trace it back. When the pipeline drops something you expected to see, you can check whether it was clustered into another finding or whether it never arrived from any agent.

To lock in the behavior, I committed synthesized reviews for six sample diffs and wrote a pipeline test that asserts the exact ranked output for a known input. That test fails if the survivor rules change order, if the LLM arbiter's clustering prompt changes in a way that affects decisions, or if a new agent gets added with the wrong priority weight.

What Phase 4 Actually Taught Me

The concurrency win from Task.WhenAll helped a lot and it took maybe an afternoon to get right. The synthesis design took considerably longer because it forced me to make explicit decisions I'd been deferring. I had to define what does "duplicate" actually mean, who wins when agents disagree, and how do I make sure I can explain any output the system produces.

If you're building a multi-agent pipeline, make those decisions before you start merging results, not after. Text deduplication feels like a quick fix until you see it eat a finding that mattered. Giving an LLM full edit access to the merged output feels powerful until you can't explain why a finding got reworded into something subtly different. The constraint I put on the arbiter is to use cluster IDs only, never touch content. That design decision I'm most confident about in this whole phase.

Phase 5 is going to be about the developer-facing output layer. Figuring out how the ranked, synthesized findings get surfaced in a PR comment in a way that's useful rather than noisy. The synthesis is solid now, so it's time to think about the last mile.