AI swarms learn when to share—and when to split the search

Learning teamwork without assigned jobs

In the November 5 preprint “s³: Simple Scaling of Swarms via Cooperative Multi-agent RL,” Kunal Jha and colleagues at the University of Washington and the Allen Institute for Artificial Intelligence investigate a practical question: can AI teams learn how to organize their work rather than follow a designer’s fixed workflow?

Their method, s³, trains teams of identical language-model copies. Each agent has its own observations, reasoning history and tools. Agents cannot inspect one another’s private work; they must explicitly send messages and choose to read incoming ones. They work asynchronously, so one agent need not wait for every teammate before continuing.

There is no assigned leader or specialist, but the setup is not instruction-free. Prompts explain the task, communication tools and general collaboration principles.

Rewarding the team’s outcome

The central training choice is a shared reward: if any agent solves the problem, the whole team benefits. Successful teams also receive a modest penalty for using more of their combined token allowance. Tokens are the chunks of text models generate, including reasoning and messages. Failures receive no reward, so quitting early cannot beat a successful attempt.

For each training problem, the researchers compare eight separate team attempts. Actions generated by relatively successful teams are reinforced, including messages that help another agent succeed. This avoids requiring the system to calculate each individual’s contribution.

Teams trained with four agents for 200 reinforcement-learning updates. Evaluation used 32 held-out problems per task and six evaluation seeds, testing populations of up to 16 agents without retraining. Comparisons matched agent-trajectory budgets and per-agent token allowances, rather than establishing equal elapsed computing time.

When everyone holds a missing piece

The first task gave each agent one private number. To succeed, at least one agent had to return everyone’s numbers in sorted order, retaining duplicates belonging to different agents. Sorting was straightforward; collecting and tracking the information was the challenge.

Using Qwen3-0.6B, s³ achieved about 93% success with four agents. A control that imitated successful teams reached about 35%, while communicating agents rewarded only for their own correct answers reached about 13%.

During training, s³ teams increasingly adopted a simple exchange: each agent broadcast its number once, early, before reading others’ messages. By the end, roughly 80% of training teams followed that pattern. Under individual rewards, agents increasingly failed to share: sending their own number helped teammates answer but did not directly help them earn individual credit.

The learned exchange remained useful beyond four agents. Its advantage over both controls persisted through 12 agents, although all methods struggled at 16. Scaling therefore exposed a remaining difficulty, not unlimited coordination.

Searching together without retracing every path

The second task placed agents in separate copies of the same unknown, branching environment. Each explored rooms through tool calls, and the first to reach the exit solved the task for everyone. The harder training and evaluation environments contained 391 rooms and 27 possible exit locations.

These experiments used Qwen3-1.7B. Importantly, the main policies first learned from scripted demonstrations: communicating explorers announced branches and reported exhausted routes, while solo explorers received matched demonstrations without messages.

After training, four-agent s³ teams solved about 90% of held-out problems. Disabling communication while retaining the shared reward reduced success by approximately 19 percentage points. Communicating teams also had nearly half the room overlap of that control. However, overlap includes corridors agents must share to reach different branches, so it is not a pure measure of wasted work.

A qualified advantage over independent attempts

A stronger comparison was best-of-N: run N independently trained agents and count success if any finds the exit. Four-agent s³ exceeded best-of-four by 9.4 percentage points, but that difference did not remain statistically significant after correction for multiple comparisons. At six agents, the 7.3-point advantage did survive correction.

Independent search approached perfect success at larger populations, leaving less room for communication to help. Shared rewards were also not essential everywhere: individually rewarded communicating agents performed similarly to s³ on exploration, unlike the private-number task.

What remains uncertain

The evidence comes from two controlled tasks, small models and one training run per condition. Six evaluation seeds measure variation in testing, not reproducibility across independently repeated training runs.

Demonstrations were crucial for exploration: without them, four-agent swarm training showed no significant advantage over independent search. Agents also sometimes repeated searches, trusted false reports or produced wrong answers despite possessing all necessary numbers. The preprint supports learned coordination in these settings, but does not establish that the approach will transfer to open-ended research or other real-world work.