AI coding agents can pass alone and fail together

An AI coding agent may understand its assignment without understanding what another agent needs from the same software. In the preprint Decentralized Multi-Agent Coordination and Communication in Shared Codebases, Rishabh Mediratta and colleagues introduce ParallACT, a benchmark designed to separate those two abilities.

The researchers study agents receiving independent requests from different developers, rather than subtasks distributed by a central planner. Their question is whether agents can produce a jointly correct result when neither initially has the complete picture.

Two kinds of collision

ParallACT contains 155 task pairs drawn from 11 open-source repositories. Each starts with a requirement based on SWE-Bench Pro, an existing software-engineering benchmark, and adds a separately motivated counterpart. The authors manually constructed 27 instances and generated another 128 with a language model, followed by automated checks and human review.

The benchmark distinguishes 70 complementary instances from 85 contradictory ones. Complementary tasks depend on a shared interface: one agent might return a list while another writes code expecting a dictionary. Their edits can merge without a textual conflict but still fail together.

“Contradictory” does not mean the users’ goals are impossible to reconcile. Instead, independently valid implementations collide, requiring a redesigned joint solution. In one Ansible example, one request changes identifier-validation rules while another adds support for dotted names. Both alter the same function, but a combined implementation can satisfy both.

The authors deliberately filtered for tasks requiring coordination. For contradictory instances, neither stacking the reference patches nor merging them with Git could produce a passing joint solution. Complementary instances required both reference implementations to pass an integration test.

A large gap between solo and parallel work

The study tested seven configurations: Claude Code with Claude Opus 4.8, plus Claude Code and Codex each paired with three open-weight models. Opus was evaluated on an 85-instance subset; the six other configurations were evaluated on all 155. Comparisons between execution strategies used identical instances within each configuration.

A single agent given both requirements served as the fully informed reference, called the “oracle.” With Opus, it passed 62% of contradictory instances and 86% of complementary ones. Two parallel agents, each unaware of the other and given only its own requirement, achieved joint success rates of 16% and 57%.

The same direction of change appeared in all six open-weight configurations. Their average success fell from 60% to 19% on contradictory tasks and from 72% to 53% on complementary tasks.

Parallel agents worked separately from the same starting repository, with their edits combined afterward. The headline results counted an instance as successful if any of three combination methods worked: patch concatenation, Git merging or a model-based conflict resolver. Even with that favorable scoring, substantial gaps remained.

Providing acceptance tests made the distinction sharper. Normally, evaluation tests were withheld. When Opus agents could debug against their own provided tests, each side’s isolated patch passed its tests in 98% of contradictory instances. Yet joint success reached only 38%, compared with 98% for one agent given both requirements and tests.

Messages are not enough

Simply telling agents that a peer existed did not significantly improve Opus performance. Messaging and explicit synchronization instructions helped in some settings, but did not close the gap.

Among messaging and synchronization variants without privileged access to the partner’s requirement, Opus performed best with Redis message inboxes and synchronization checkpoints. It reached 32% on contradictory instances and 77% on complementary ones. Its effective token cost on contradictory tasks was 2.23 times the single-agent reference. This measure weights different token categories; it is not a direct dollar bill.

A separate “communication oracle” gave each parallel agent its partner’s requirement and instructions to communicate. It achieved higher success rates of 56% and 83%, respectively, compared with the single-agent oracle’s 62% and 86%.

Analysis of Opus communication logs suggested a timing problem. About one-third of messages were never read, and only 21–23% were read before the recipient’s first edit. Agents often discussed the relevant interface or contested code, but agreements did not reliably translate into compatible changes.

Taking turns worked better. A single handoff, allowing the second agent to see the first’s edited files, raised Opus joint success to 54% on contradictory tasks and 77% on complementary ones at roughly the token cost of unaware parallel execution. Adding a second turn for each agent increased costs without a significant success improvement. The authors argue that enforcing turn-taking would become impractical at large scale.

These findings are diagnostic, not an estimate of everyday failure rates. The benchmark deliberately selects interacting tasks, examines only agent pairs and does not establish what happens in larger teams or longer workflows. Within those boundaries, it shows why working implementations are not enough: agents must also discover shared assumptions, divide responsibility and act on one another’s information.