SlackBench tests whether AI agents can piece together a collaborative project

A workspace built for evaluation

Mingxuan Li and colleagues present SlackBench as a test of collaborative project understanding rather than isolated document retrieval or code repair. Their argument is that workplace answers often depend on connecting conversations, repository history and evolving decisions. This is the authors’ motivation for the benchmark, not evidence that it captures every form of knowledge work.

The environment contains 15 Slack channels, roughly 3,000 messages, 14 GitHub repositories and nine Google Docs. It is not an untouched archive of private organizational conversations. Humans and coding agents carried out research projects in a live Slack workspace, producing discussions tied to actual project artifacts. The authors describe this process as privacy-preserving reenactment and simulation.

About 35.0% of messages were directly human-authored, and another 35.2% were substantially rewritten by humans. Manual cleanup included correcting hallucinated content against repositories and replacing real API keys with fake ones. These interventions help explain how the environment was constructed, while distinguishing it from naturally occurring workplace records.

Four kinds of challenge

SlackBench’s 140 queries cover four capabilities:

• Agentic search: 58 tasks. Find information across conversations and artifacts, sometimes tracking changes over time.
• Hallucination avoidance: 48 tasks. Recognize missing information, false premises or answers outside a specified time window, and explain the problem.
• Goal-directed summarization: 14 tasks. Assemble a faithful account for a purpose such as describing a researcher’s contributions.
• Circumstance inference: 20 tasks. Identify behavioral patterns distributed across long conversations.

The final category needs particular caution. For each query, the researchers modified the workspace to embed a predetermined pattern—for example, by reassigning substantive contributions and adding vague messages from a designated participant. Success therefore measures recognition of constructed signals, not validated judgment about real employees’ motives or character.

What the experiments report

The main results compare 33 model-and-agent-harness combinations, using 24 models and eight harnesses. A harness is the software surrounding a model that manages tools and execution. Main experiments used medium reasoning effort and a 20-minute limit per query, with category results averaged over three runs.

The authors report the highest overall score, 80.25%, for Antigravity with Gemini 4 Argon. Most evaluated combinations scored below 66%. These are benchmark-specific results, not general workplace reliability ratings. Overall scores give equal weight to the four categories, despite their different numbers of tasks; summarization also receives partial credit through checklists.

The failure analysis illustrates why retrieval alone is insufficient. Agents sometimes found later evidence but retained an outdated conclusion, treated a proposed document section as completed work, or inflated someone’s role when writing a polished contribution summary. Other failures involved answering a question without correcting its mistaken attribution.

Access design changes performance

In seven paired comparisons, giving agents direct terminal access to exported Slack files produced higher overall scores than access through Model Context Protocol tools. For example, Codex CLI with GPT-5.5 rose from 45.52% to 59.90% overall.

The authors associate file access with more direct reading and less metadata-heavy searching. However, higher overall scores did not mean improvement in every category: some search and hallucination-avoidance scores fell. The finding supports examining interface design alongside model choice, rather than assuming terminal access is universally superior.

Limits on the conclusions

Answers were graded by GPT-5.4 against human-curated references and rubrics. A manual review of 264 judgments for one agent configuration found 95.8% agreement with human judgment. That is useful validation, but it does not establish error-free grading across every system.

Three independent researchers achieved an average overall score of 84.5%, with a reported 95% bootstrap confidence interval of 77.0%–92.0%. They were allowed to use AI tools, so this is not an unaided-human baseline or clear evidence of a precise human–agent gap.

The study is a preprint, not established peer-reviewed evidence. Its academic research setting, reconstructed conversations and deliberately engineered social patterns constrain generalization. The authors also did not test alternative prompting strategies. SlackBench offers a concrete way to probe evidence reconciliation and faithful answering; its results do not establish readiness for autonomous workplace deployment.