Cohere's Coding-Agent Study: Where Cheap Workers Win
Cohere's 200-task study finds planning-only delegation can match all-stage coding-agent teams at roughly half the inference cost. See the data and a pilot checklist.
Cohere's October 14, 2026 research challenges a common assumption about coding-agent systems: giving every stage more AI workers is not automatically the best use of money. On its ProgramBench experiments, assigning workers to exploration and planning produced roughly the same average result as delegating planning, coding and verification together, at about half the inference cost. A strong lead model paired with cheaper planning workers got close again for substantially less.
The useful takeaway for developers is a routing decision, not a universal savings promise: identify the bottleneck in a task before deciding which agents to hire, which model should lead, and when to stop delegating.
What Cohere actually tested
Researchers ran a minimal shared-workspace agent harness on ProgramBench, a set of 200 software-rebuilding tasks. Each task supplies a reference executable and documentation but not the original source code. Agents must explore behavior, implement a replacement and pass hidden behavioral tests. Cohere compared solo agents, teams with blocking or asynchronous leads, delegation at different workflow stages, stronger versus cheaper workers, and different team sizes. Most main configurations were repeated three times on all 200 tasks.
The paper reports average reward as the percentage of behavioral tests passed, not the share of entire applications delivered successfully. Its dollar figures are measured inference costs for the tested workloads. Neither number alone measures production readiness, maintenance quality or security.
The three cost-quality comparisons that matter
Selected ProgramBench configurations reported by Cohere
| Team configuration | Average test-pass reward | Reported inference cost | What changes |
|---|---|---|---|
| Sol lead and Sol workers, all stages delegated | 63.2% | $510 | Higher-cost comparison baseline |
| Sol lead and Sol workers, planning delegated only | 63.6% | $249 | Similar reward, roughly half the cost |
| Sol lead and 16 Luna planning workers | 61.8% | $155 | About 70% lower cost than all-Sol all-stage, 1.4 points lower reward |
These are separate configurations in the study, not guaranteed interchangeable production settings. In particular, the $155 versus $510 comparison is a roughly 69.6% reduction in reported inference cost, not a 70% reduction in engineering salaries or total cost of ownership. The 61.8% and 63.2% figures describe mean test-pass reward, not the percentage of tasks fully completed.
Why planning won on this benchmark
ProgramBench agents must discover a black-box program's behavior before they can reproduce it. Multiple planning workers can probe different interfaces and edge cases in parallel, giving the lead model a better map of the target. Once that information is assembled, the lead can implement and check the program without multiplying every coding and testing step.
That advantage does not transfer automatically to work with a complete specification. In Cohere's smaller Commit0-Lite comparison, agents received repository skeletons, documentation and tests. Delegating implementation was the strongest stage in eight of ten tested team setups. The authors caution that this secondary benchmark is small and already near saturation for strong solo agents.
A routing guide for a real coding-agent workflow
Make Better's interpretation: allocate agents to the current bottleneck
| Task signal | Try delegating | Keep with the lead or reviewer |
|---|---|---|
| Unknown legacy behavior, sparse documentation | Independent exploration, test discovery, interface mapping | Architecture decision and integration |
| Clear API contract and reliable tests | Independent implementation units with explicit boundaries | Integration and acceptance criteria |
| High-risk change or unclear correctness | Adversarial review and independent checks as an additional control | Final approval and production deployment |
| Many workers but flat quality gains | Reduce worker count; benchmark smaller models | Measure marginal value before scaling |
This routing guide is editorial advice informed by the research, not a tested universal policy. For example, a migration with clear unit tests may benefit from implementation parallelism, while an undocumented integration may benefit more from discovery workers. Always preserve independent review for consequential changes, even when a benchmark makes verification delegation look less cost-effective.
A small pilot before scaling your agent team
- Collect 15–30 representative engineering tasks and define measurable acceptance tests; include both unknown-behavior and clearly specified work.
- Run a strong solo-agent baseline, then test planning-only delegation and all-stage delegation with the same task inputs and permissions.
- Measure successful acceptance tests, human fixes, elapsed time, token and tool spend, and any unauthorized or unsafe actions.
- Try cheaper workers only for bounded exploration subtasks; keep a stronger lead responsible for synthesis and final implementation.
- Calculate cost per accepted task and stop increasing worker count once quality gains no longer justify the added cost.
What the evidence cannot tell you
Cohere's results are an internally reported controlled study. The benchmark intentionally emphasizes rebuilding behavior from an executable, which makes exploration valuable. Different repositories, prompts, agent frameworks, model prices and review requirements can reverse the ranking. The authors also observed that model families delegated differently, and their additional five-model comparison used only one run per configuration on a 30-task subset. Small differences there may be noise.
For an enterprise pilot, treat the published results as a reason to test cheaper planning workers—not as a procurement forecast. Add tool, infrastructure and human-review costs to the inference bill, and inspect failure severity rather than optimizing an average score alone.
Questions people ask
Do teams of coding agents always outperform a single coding agent?
No. Cohere's October 14 research found meaningful gains for some agent-team configurations on ProgramBench, but those teams often consumed several times more inference budget. On a separate 30-task model comparison, several models saw little or no team benefit. Start with a strong solo-agent baseline, then compare completion quality, cost and elapsed time on your own tasks before adding workers.
When should a coding-agent team delegate planning instead of implementation?
Delegate exploration and planning first when the specification is unclear and agents must discover how existing software behaves. That was the bottleneck in Cohere's ProgramBench tests. When requirements and unit tests are already supplied, the study's smaller Commit0-Lite evaluation favored delegating implementation in eight of ten configurations. Choose the delegation stage based on what your task actually lacks, not a universal planning-first rule.
How much did cheaper planning workers save in Cohere's benchmark?
In one reported ProgramBench configuration, a GPT-5.6 Sol lead with sixteen Luna planning workers cost $155 and achieved 61.8% average reward. An all-Sol team delegating every stage cost $510 and achieved 63.2%. That is about 70% less reported inference cost for a 1.4-point lower benchmark reward. These are experimental aggregate figures, not guaranteed per-task API bills or production savings.
Will adding more worker agents always improve coding results?
No. Cohere saw planning-only teams improve from 59.6% to 61.8% reward as cheaper workers increased from two to sixteen, but implementation and verification teams peaked around four workers and then slipped. More agents can also increase coordination and token costs. Measure marginal quality gain and cost per correctly completed task, and stop adding workers when the benefit levels off.
The best coding-agent team may be the one that delegates less. Cohere's ProgramBench study shows that a strong lead with narrowly assigned, lower-cost planning workers can approach an expensive all-stage team. Find your task's bottleneck, benchmark against a solo agent and optimize for accepted work—not agent count.
Sources & useful resources
- Cohere: Lessons from scaling teams of coding agents (Oct 14, 2026)— Primary controlled experiment, all reward and cost comparisons and secondary Commit0 results
- ProgramBench: Can Language Models Rebuild Programs From Scratch?— Original benchmark design and behavioral-test scoring
- Commit0: Library Generation from Scratch— Original specification-and-unit-test benchmark design