What Building a Six-Agent AI Code Reviewer Taught Me About Agentic Systems

I Built a Six-Agent GitHub Code Reviewer for Free — Then It Started Calling Its Own Tools 43 Times

I wanted a portfolio project that actually demonstrated multi-agent design — not another wrapper around a single LLM call with a nice UI on top. So I built PR Review Crew: a six-agent GitHub pull request reviewer, orchestrated with ChatDev 2.0 (OpenBMB's zero-code multi-agent platform), free and open source under MIT.

Point it at any pull request. Six agents fetch the diff, review it from three angles in parallel, synthesize the findings into one comment, and post it back to GitHub. Here's what that looks like end to end, and the four real bugs I hit trying to make it work on free-tier models — bugs that taught me more about running agentic systems in production than the parts that went smoothly.

The architecture

Six agent nodes, defined in one YAML graph:

Diff Fetcherparses- owner/repo#123out of the task and calls a- get_pr_filestool to pull the changed files and patches.
Logic & Style,- Security, and- Docs & Testsreviewers run in true parallel off the same diff, each reviewing from one lens and ignoring the others.
Synthesizerwaits for all three, then merges their findings into one Markdown comment with a verdict.
Comment Postercalls a- post_review_commenttool to actually publish it.

Two design decisions turned out to matter more than I expected once I started testing against real models.

The first is context propagation. ChatDev's graph only forwards each node's output to its direct downstream edges by default — so by the time you're four hops downstream, you've lost the original task. Rather than relying on some implicit shared state, every node in the crew echoes a REPO: / PR: header forward as an explicit protocol. It's more verbose than magic context-sharing, but it's legible: you can read any single node's output and know exactly what it's reviewing.

The second is separating deterministic actions from reasoning. Fetching a diff and posting a comment aren't judgment calls — but ChatDev's python node type only runs pre-existing scripts from a shared workspace, it doesn't take inline code in the YAML. So Diff Fetcher and Comment Poster are agent nodes too, just with a single bound tool, temperature pinned to 0, and a tightly scoped instruction. They're "agents" in name only; in practice they're deterministic steps wearing an LLM costume.

Bug one: the loop nobody told me about

The first real bug showed up testing against Groq's free tier — llama-3.3-70b-versatile, picked to keep the whole thing free to demo — in Comment Poster, and it wasn't in my prompt or my Python. It was in an undocumented engine behavior. Every agent node in ChatDev gets an implicit self-loop edge: after a tool call, the engine feeds the result back into the same node and calls the model again, so it can decide whether to call more tools or finish. The loop only releases downstream once a response comes back with no tool calls.

That's a sensible default. It's also exactly what let Comment Poster call post_review_comment successfully, then look at its own success message, apparently decide "well, better call it again to be sure," and keep looping — with the context window growing every iteration. One run made 43 separate calls to Comment Poster before I caught it. Nothing in the base engine stops a model from re-invoking a tool it already used correctly.

The fix wasn't code, it was the prompt. I had to explicitly teach Comment Poster to recognize its own past success and stop:

"If the conversation already contains a result from ``post_review_comment... the tool has already succeeded. Do NOT call it again for any reason."

Obvious in retrospect. Not obvious until I watched a runtime log with the same [DRY RUN] Would have posted... message repeating with a growing token count.

Later, digging into the actual SDK source instead of just the logs, I found the mechanism itself: a method called _handle_tool_calls in ChatDev's agent_executor.py, built around one while True loop. Each pass appends whatever the model just said to the conversation, returns immediately if that response made no tool calls, and otherwise executes the tool calls, appends their results, and calls the model again with the now-longer conversation — repeating until a clean response arrives or a configurable tool_loop_limit is hit. None of that is ChatDev-specific. Every tool-calling agent loop, in any framework, has this shape: call the model, check the response for a tool call, run it if there is one, feed the result back in, call the model again. The only thing that ends the loop is a response with no tool call in it — which is exactly the gap Comment Poster fell through. If you're building or debugging one of these, that's the single line worth finding in whatever framework you're using: what actually ends the loop.

Bugs two through four: the free-tier gauntlet

That runaway loop is almost certainly what burned through llama-3.3-70b-versatile's 100,000 token daily quota — 43 unnecessary calls in a single run adds up fast. llama-3.1-8b-instant draws from a separate quota, so once the loop was fixed, moving to it was what actually unblocked me. It opened a new front, though: small, fast models are worse at precisely the things a tool-calling pipeline depends on.

The emoji that broke JSON. Everything worked until the Synthesizer's verdict line included a 🟡. In JSON, that emoji has to be escaped as a UTF-16 surrogate pair: 🟡. When Comment Poster's model had to reproduce that string inside its own tool-call arguments, it garbled it to \ud83d\dda5 — close, but \d isn't a valid JSON escape sequence. Groq's server-side parser rejected the whole tool call before it ever reached my Python. The fix wasn't a longer prompt or a retry — it was removing the emoji from the pipeline entirely. Minor changes suggested, no 🟡, no problem. Small models are unreliable at reproducing exact escape sequences for characters outside the basic multilingual plane; the lesson generalizes past emoji to any Unicode-heavy content flowing through a tool-call argument.

The token budget that was too honest. Comment Poster's max_tokens was set to 500 — reasonable if it only needed to emit a short tool call and a short confirmation. But the tool call has to carry the entire synthesized review as a JSON string argument. A ~3.4KB Markdown body blew straight through 500 tokens mid-generation, and the model's JSON got cut off before it closed its braces. Groq rejected the truncated result the same way it rejected the malformed emoji — a tool_use_failed error with a failed_generation field showing exactly where the string just... stopped. Raising max_tokens to something that actually accounted for the payload size fixed it.

The tool result that echoed too much. The last bug was the subtlest. Even with the emoji and token issues fixed, one run hit a 413 Request too large on Groq's tokens-per-minute limit — a hard cap of 6,000 for that model, and the request asked for 6,433. The culprit: my post_review_comment tool's dry-run branch echoed the entire review body back as its return value, up to 4,000 characters. Once the self-loop fired and reassembled context for the next turn, that body existed twice — once in the original input, once in the tool result — pushing a request that should've been comfortably under the limit over it. Trimming the echo to a 200-character preview cut the duplication and brought the request back under the cap.

None of these three bugs would have shown up against a larger, more capable model. They only surfaced because I was running a real tool-calling pipeline against a genuinely resource-constrained model — which, if you're building anything meant to run affordably at scale rather than as a demo with an unlimited budget, is exactly the environment you should be testing in.

Bug five: the model that reasoned itself into silence

I came back to this a few days later to build a more substantial demo — a real merged Flask pull request (pallets/flask#6096, a small IPv6 host-parsing fix) instead of a one-line placeholder — and the crew broke immediately. Groq's free-tier lineup had rotated entirely in the interim: llama-3.1-8b-instant and llama-3.3-70b-versatile were both gone, replaced by a set of reasoning models (openai/gpt-oss-20b and -120b, qwen3.6-27b, groq/compound). Swapping in gpt-oss-20b got past the 404, but the Logic & Style reviewer came back empty, while Security and Docs & Tests succeeded on the identical diff.

Claude Code flagged it while walking through the output: 1200 output tokens spent, zero characters of visible content. A direct call to Groq's API with the same prompt showed why — finish_reason: "length", 1198 of the 1200 tokens billed to a reasoning field, content: "". gpt-oss-20b emits its chain-of-thought as a separate channel before the actual answer, and that channel draws from the same max_tokens budget as the answer itself. Logic & Style's prompt — find bugs, edge cases, naming, dead code — apparently invites more deliberation than a security or docs pass, so it was the one node that ran out of budget before writing anything down.

The first fix was one number: raising max_tokens from 1200 to 3000 on the three reviewer nodes, confirmed against Groq's API directly before touching the YAML. It held on the flask re-run. It didn't hold on the next one.

To get an actual posted comment — not just a dry run — I built a second small repo, pr-review-crew-demo, seeded with a real (if toy) pull request: a path-traversal bug, a couple of missing tests, no doc updates. Same three reviewer nodes, same max_tokens: 3000, different diff — and Logic & Style came back empty again, this time burning the full 3000 on reasoning instead of 1200. Comparing runs at that same max_tokens: 3000 ceiling showed the real shape of the problem: reasoning-token usage for the identical prompt ranged from about 1,100 tokens up to the full ceiling itself, run to run — sometimes leaving plenty of room for an answer, sometimes leaving none. Raising the ceiling had just been buying better odds, not a guarantee. reasoning_effort: "low" fixed it properly — it caps the model's own chain-of-thought rather than hoping the budget outruns it, and dropped reasoning usage to under 20 tokens across every retest. It's a parameter specific to reasoning models; pointed at a non-reasoning model like plain gpt-4o-mini, the same call would just fail, so it's documented in the YAML as something to strip out if you switch model families.

Getting that comment to actually post surfaced one more thing, unrelated to any model. post_review_comment failed with a 403 and "Resource not accessible by personal access token" — not a scope problem but a repository problem. The token in .env was a fine-grained GitHub PAT, which grants access to a fixed list of repos chosen when the token was created, and pr-review-crew-demo didn't exist yet at that point, so it wasn't on the list. The fix was adding the new repo to that list, not switching token types. A classic token with repo scope would have sidestepped the problem too, since it covers every repo you own, present and future — but that's a much wider blast radius for a script that only ever needs to comment on pull requests. Fine-grained, and re-scoped by hand as the list of repos grows, is the right tradeoff here. The 403 just doesn't tell you which kind of wrong you're dealing with — worth checking the token's repo list before reaching for a broader one.

The throughline

Every one of these bugs traces back to the same root cause: small, fast models need leaner payloads at every single hop, not just shorter system prompts. It's tempting to think of prompt engineering as the whole surface area of working with LLMs, but a tool-calling agent pipeline has payloads flowing in several directions — into the model, out as tool-call arguments, back in as tool results, forwarded to the next node — and each one is a place a large or awkward blob of text can quietly break something two or three steps downstream. Bug five inverted the direction but not the shape of the lesson: a reasoning model's token budget has to cover the thinking you never see, not just the answer you do.

The other lesson was more about the tooling than the models: ChatDev's implicit self-loop edge is a real, load-bearing piece of engine behavior that isn't obvious from reading the docs or the YAML schema. It only became visible by actually running the workflow and reading the raw event logs line by line. If you're evaluating any multi-agent framework, budget time for exactly that — not just reading the reference docs, but watching what the engine actually does when a tool call comes back.

The same crew, running in DevAll's web console

The CLI is fine for a demo, but PR Review Crew is a ChatDev workflow, and ChatDev 2.0 ships a full web console — DevAll — for building and running graphs like this one visually. I wanted to see if the crew held up outside the terminal. The chatdev pip package run_review.py depends on only ships the backend; the visual canvas is a separate Vue frontend that isn't part of the SDK install, so Claude cloned the actual ChatDev repo, stood up both servers locally with plain pip/venv since I don't have uv installed, and pointed it at a copy of pr_review_crew.yaml.

DevAll rendered the graph straight from the YAML with no extra work — same six nodes, same fan-out shape, matching the mermaid diagram in the README. Clicking a node opens its live config: prompt, provider, model, right there instead of scrolled deep in a YAML file.

Getting an actual run to complete surfaced a real difference between DevAll and the CLI. run_review.py patches every node's provider and model at runtime from --model/--base-url, which is why the YAML file itself can keep OpenAI's gpt-4o/gpt-4o-mini as placeholders. DevAll's Launch tab has no equivalent — it runs the YAML's node configs exactly as written. Pointed at Groq with the placeholder names still in the file, every node failed with the same model_not_found error I'd already solved once for the CLI. Fixing it meant hardcoding openai/gpt-oss-20b into the YAML itself, the same reasoning model from Bug five.

With that fixed, launching against pallets/flask#6096 again — the same PR from that bug — worked, with one more free-tier wrinkle along the way. The three reviewers fire in parallel, and one, Logic Style Reviewer, came back with a 429: Groq's tokens-per-minute budget is shared across the org, not per request, and the other two reviewers' calls had already spent most of it before Logic Style's request landed. The engine's retry logic — the same self-loop mechanism from Bug one, doing what it's supposed to this time — waited out the window Groq specified and retried; the node finished about 33 seconds later, and the run completed end to end down to the dry-run comment preview.

Claude drove the setup, caught the model-mismatch error, and read the rate-limit response closely enough to explain why it happened before I asked. I made the calls that mattered — sticking with the free Groq tier instead of switching to a paid one to make the demo prettier, and running it against the same PR from Bug five instead of a fresh one, so the two pieces would actually compare.

The interface changed; the mechanism didn't. The same implicit retry loop that let a bug slip through in Comment Poster is what quietly absorbed a rate limit here — a friendlier UI just means you get to watch it happen instead of reconstructing it from logs afterward.

Try it yourself

PR Review Crew is on GitHub, MIT-licensed, and designed to demo in about a minute: point it at any public PR with --dry-run and watch the whole crew work without posting anything. The README has setup steps and free-tier recipes for Groq, Gemini, and Ollama if you don't want to touch a paid API key.

For an example of it actually posting, not just printing, pr-review-crew-demo is a small sandbox repo built for exactly that — the review it left is real, not simulated.

If you build on it — a fourth specialist reviewer, a different git host, a human-approval gate before anything posts — I'd genuinely like to see what you do with it.