Does a Team of AI Coding Agents Beat One? A Bay Area Engineer Built the Tool That Keeps the Receipts

SAN FRANCISCO, Nov. 4, 2026 — Ask a developer whether three AI coding agents working together will fix a bug better than one, and the honest answer is usually a guess. Orchestration tools that run agents in parallel have multiplied over the past year. Some have collected more than 100,000 stars on GitHub. Very few report what a team of agents actually delivers, or what the extra agents cost.

Bicheng (Kenneth) Wang, a Bay Area engineer who builds machine-learning decision systems, decided the question should be measured instead of assumed. His open-source project multi-agents runs a team of coding agents on a task inside a developer's own repository, returns one patch, and attaches an itemized record of every agent call: tokens, dollars and seconds. A built-in benchmark is designed to compare each team with a single agent on GitHub issues created after the models' training cutoffs.

The project's roadmap states the gap it is aimed at in one sentence: dozens of tools run several coding agents in parallel, and "none of them tells you whether the team beat a single agent, or what the extra agents cost."

Released on the Python Package Index on Oct. 20 as multi-agent version 0.1.0, the project had 136 stars and 12 forks on GitHub as of Oct. 27, 2026, up from 136 stars on Oct. 26.

The evidence points both ways

The case for agent teams sounds intuitive. One agent writes the code, another reviews it, a third picks the best attempt. In practice, published research disagrees with itself.

A study of 260 agent configurations across six benchmarks, "Towards a Science of Scaling Agent Systems" (arXiv:2512.08296), found every multi-agent architecture it tested, averaged across models, between 2.1 and 14.9 percent worse than a single agent on a 20-task sample of SWE-bench Verified, a standard coding benchmark, while the same study found large gains on tasks that split naturally into parallel parts. Other teams reported improvements. Agyn (arXiv:2602.01465), a design with a coordinator, researcher, engineer and reviewer, reached 72.2 percent on SWE-bench Verified with different models in different roles, 7.2 points ahead of the mini-SWE-agent baseline but only 0.4 points above the strongest single-agent baseline in its comparison. A decentralized design that pairs a patch editor with a validator and adds an explorer for vaguely described issues, icat-agent (arXiv:2606.25514), gained 3.6 and 8.4 points over mini-SWE-agent.

The benchmark behind many of those numbers has its own problem. OpenAI stopped reporting SWE-bench Verified in February 2026, citing contamination and flawed tests: models may have seen the answers during training.

After building the benchmark, Wang surveyed how the most-starred agent projects organize their teams. The survey, dated Oct. 18 and published in the repository, covers company-style control planes, company-simulation frameworks and coding-team orchestrators, and tags every claim by where it was verified: in source code, in documentation, or only claimed in a README and not verified. Its conclusions are blunt. On coding tasks, a hierarchy alone does not beat one agent. The gains that do appear come from two places: independent verification by an agent with its own isolated context, and parallel work that is merged only after it passes a test gate. And none of the sources it reviewed, the survey notes, compares the same model as a single agent and as a team with the reviewer switched on and off, on SWE-bench-style issues.

That missing comparison is what multi-agents is built to produce.

Four ways to staff a task

A developer picks a team with a short specification string. There are four kinds.

solo runs one agent. It is the baseline every other team is measured against.

review pairs a coder with a reviewer. The reviewer scores the patch in a separate copy of the code and sends feedback, and the coder resumes its session and revises. The team returns the best-scoring patch, not simply the last one.

best runs several workers in parallel, each in its own copy of the repository and possibly on different tools and models, and a judge picks the strongest patch.

plan has a planner split the task into independent subtasks. Workers solve them in parallel, the patches are merged, and an integrator resolves conflicts and gets one fix pass if the check fails. Subtasks that fail or change nothing go back to the planner for another round, handed to a different worker when the team has more than one. Each subtask can optionally run as its own coder and reviewer loop.

The workers are tools developers already use: Anthropic's Claude Code, OpenAI's Codex, or a built-in agent that can call any model through the OpenAI Agents SDK. The command-line tools run on the developer's existing login, so no API keys are needed for them.

Isolation is the design principle underneath. Every worker gets its own git worktree, a separate checkout of the same repository, so each agent edits its own copy rather than another agent's or the developer's files. The working tree is left untouched unless the developer passes --apply.

A receipt for every agent call

What separates multi-agents from most orchestrators is the accounting. The README prints the real output of a two-worker best team, one Claude Code worker on Claude Haiku 5.5 and one Codex worker on GPT-6 Luna, fixing a failing test in a small repository. Each worker and each judging call appears as its own line, with input, cached and output tokens, cost and time. The whole run read 729,000 input tokens, 604,000 of them from cache, wrote 8,500 output tokens, and cost $0.0342 over 85 seconds. A plan run on a repository with two unrelated bugs split the work into two subtasks, solved one with each tool, passed the project's tests, and cost $0.0212 over 26.8 seconds.

The rules for those numbers are strict. Costs come either from the tool's own report or from list prices with a cited source. A run that cannot be priced is shown as unpriced and makes the total unpriced; the documentation says "nothing is guessed." A --max-cost flag stops new agent runs once a dollar limit is spent, and the README is careful about what that does and does not promise: runs already in flight finish, so the total can overshoot by their cost. Because an unpriced run cannot be counted against a budget, the tool refuses to start a budgeted team that includes an unpriced worker unless the developer explicitly accepts that those runs fall outside the limit.

Each run's full record is saved as a JSON file outside the repository, so the cost of a team is something a developer can audit later, not just a number on screen.

From GitHub issue to draft pull request

Recent work has moved the tool from experiment toward everyday use. A --check option runs the project's own tests, build or linter against the final patch in a separate worktree and records whether it passed; with an AI judge, candidates that fail the check rank below those that pass. The check can also stand in for the reviewer or judge entirely, giving a test-driven loop with no language model in the reviewing seat.

--issue takes the task straight from a GitHub issue, and --pr commits the result to a new branch and opens a draft pull request with the run summary attached, without touching the developer's checkout. The README flags the security boundary that comes with it: issue text is untrusted input, so on public repositories it should not be combined with the tool's permission-bypass mode or with the built-in SDK agent, which has no permission system. By default each coding tool runs inside its own sandbox, with edits and test commands confined to its worktree.

The repository also works as a plugin marketplace for both Claude Code and Codex. Once installed, a developer can ask the agent they are already using to hand a task to a team, review the summary and the patch, and apply it only after approving.

A benchmark designed against contamination

The benchmark is where the project's central question gets answered. It draws tasks from SWE-bench-Live MultiLang, a dataset of real GitHub issues in Rust, Java, Go, C, C++, C#, JavaScript and TypeScript. The project's documentation explains the choice: it is the only candidate it found that was both fresh and fully self-describing, with each task carrying its own container image, test commands and log parser. Researchers can keep only issues opened after a given date. With Anthropic documenting a June 2026 training cutoff for its current models, filtering to issues created from July 1, 2026 leaves 231 tasks from 57 repositories. The documentation notes that OpenAI's cutoffs for its newer models have not been checked yet.

Each task runs in a fresh sandbox, either remote or on a local Docker daemon, with git history stripped so an agent cannot read the original fix from the repository. Patches are graded with the dataset's official hidden tests, through a port of the upstream grading code. The report gives each team's resolve rate with 95 percent confidence intervals, paired wins and losses against the solo baseline on the same tasks using an exact McNemar test, cost per task and per resolved task, and wall time. A team that runs out of time is graded on whatever patch it left rather than dropped, so a slow team cannot shrink the comparison down to the easy tasks.

The documentation is equally direct about the benchmark's limits. In the planned 50-task pilot, only large effects reach statistical significance: six wins against no losses gives a p-value of 0.031, while a gap of 4 to 8 points between a team and a single agent would not be detectable. It also prices the experiment in advance. The recommended first pilot, which compares the review and best teams with a single agent on low-cost models and adds a smaller spot-check on a frontier model, is budgeted at about $467.

What is finished, and what is not

The project publishes a status table rather than a feature list, and it is specific. The team runner with all four designs, the Claude Code and Codex integrations, the test check, the issue-to-pull-request flow, the plugins and the run budget are marked working and smoke-tested with the real tools. The any-model SDK agent is implemented and tested offline but has not yet run against a live model. The benchmark is built and tested offline, and grading and agent workspaces on a local Docker daemon have run live; a team solving a benchmark task, and anything on remote Modal sandboxes, has not. The results row reads: pending the first pilot run.

That candor is consistent with the rest of Wang's open-source work. His ai-decision-system produces replayable records for AI-assisted decisions, and his RL-pricing lab re-tests published claims about trading AI, starting with his own 2021 paper, whose headline return it found does not hold under stricter rules. multi-agents applies the same habit to the most hyped claim in AI-assisted software development: that more agents means better code.

Why it matters

Coding agents are moving from autocomplete to autonomous work on real repositories, and teams of agents are the next step many tools are selling. For an engineering manager deciding whether to pay for three agents instead of one, the useful numbers are simple: how often the team fixes the issue, how often the single agent does on the same issues, and what each costs. Until now those numbers have mostly come from vendors or from benchmarks that models may have memorized.

multi-agents is free, released under the MIT license, and installable with a single command. Its contribution is a method anyone can rerun: fresh tasks, isolated workers, hidden tests, paired comparisons and a priced line for every call. When the first pilot results are published, they will come with the receipts.

About Bicheng (Kenneth) Wang

Bicheng (Kenneth) Wang is a Bay Area engineer who builds machine-learning decision systems for digital commerce, and the methods used to evaluate and control them. His independent open-source work includes multi-agents, a runner and benchmark for teams of coding agents; ai-decision-system, replayable records for AI-assisted decisions; RL-pricing, a reproducibility lab for trading AI; and financial-agent, a research and paper-trading platform with gated execution. He is the co-author, with Xinyi Zhang, of "Deep Reinforcement Learning for Equity Trading: Benchmarking Actor-Critic Methods with Forward Retraining" (arXiv:2609.31870).

Learn more: bichengwang.github.io · github.com/BichengWang/multi-agents