TRANSMISSION // 2611.23142v1
Can AI Agents Help Keep Software Documentation Current?
Source published
A 43-day pilot used AI agents to find documentation gaps in support tickets and propose fixes. Results suggest repairs can improve later answers, but the study leaves important limits.

Software changes can outpace the guides customers rely on. A deployed system called TriAgent tried to spot that mismatch by comparing its documentation-based support answers with human responses, then routing possible fixes to human reviewers.
The documentation gap
When software changes quickly, its user guides can fall behind. That creates a problem not only for people looking for help, but also for AI assistants that treat documentation as an authoritative source. If the guide is incomplete or outdated, an assistant may confidently repeat the gap.
In Mind the Gap: A Deployed Multi-Agent System for Documentation Maintenance, Lorenzo Masoero describes a system designed to turn support work into a feedback loop. The core idea is simple: compare what an AI assistant says using existing documentation with what a human support engineer says. When the human adds information the documentation does not cover, that difference can point to a gap worth fixing.
How the system worked
The system was tested on the support queue for a single A/B testing analysis service. Its agents handled different stages of the process: TriAgent drafted documentation-grounded replies; other agents examined human responses and tested documentation coverage; and a Documentator drafted changes. Proposed documentation fixes passed an automated self-review and then required human approval before merging. The agents did not edit the product’s code.
The paper reports a 43-day shadow-mode pilot. In this setup, AI responses were available to support staff, but the study does not describe the system as independently sending answers to customers. The system triaged 499 tickets and identified 92 documentation gaps. Some gaps came from differences between AI and human responses; others came from retrospective review of resolved tickets or synthetic questions designed to probe the knowledge base. A code-change watcher was also included, but it detected no gaps during the pilot.
What the evaluations found
The researchers extracted 1,121 factual claims from 192 AI responses that could be compared with human resolutions. The human response did not address most claims, so those were classified as unverifiable rather than correct. Of the 302 claims the human response did address, a panel of three language-model judges found 245 correct and 57 incorrect—81.1% correct among these verifiable claims. That figure does not establish the accuracy of every claim the system made: 72.5% of claims were unverifiable under this comparison method.
The judges also compared the AI and human replies. Among pairs of similar length, they preferred the AI reply in 70.5% of cases and the human reply in 23.2%; the rest had no majority verdict. Response length matters: the authors report that the AI’s overall preference was partly associated with its tendency to give longer answers. The two sources also contributed different information. Human replies often included operational details absent from the documentation, and those additions could become signals for a documentation update.
To test whether a repair helped, the researchers compared an agent’s answer before a documentation change with its answer to the same ticket afterward. In 35 selected cases—where the earlier answer was not rated high-confidence—judges preferred the post-repair answer 71.4% of the time. A blinded human audit supported this before-and-after result. This is evidence that the updated documentation helped on those repeated questions, not proof that every fix will improve answers to other questions.
What remains uncertain
The findings come from one service and one 43-day pilot, so they may not transfer directly to services with different users, documentation, or support practices. The automated evaluations rely on language-model judges and human resolutions as the reference; a human audit found only fair agreement on the correctness and usefulness assessments. The authors therefore treat those headline rates as panel-based estimates, while describing the before-and-after improvement as human-validated. That audit used one annotator, not a larger group.
The improvement test was also limited: it covered 35 selected gaps and re-answered the original triggering ticket. Whether fixes help with new, related questions was not measured. And although drafting a fix took under five minutes, the study did not measure how much time human review and merging required, so it cannot show whether the process saved staff time overall. Every change still required human approval.
The paper argues that support tickets can act as a practical signal for improving documentation. Its pilot offers evidence that this feedback loop can surface gaps and improve some later answers. But broader deployment, reliable proactive detection, and the overall workload for reviewers remain open questions. The source is an arXiv preprint; readers should treat the results as research findings from a limited pilot, not as established evidence of general performance.
- AI agents
- software documentation
- documentation maintenance
- support tickets
- large language models
- human review
Related reading
Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.
Turning AI agent traces into reusable workflows
Source published
Shared keywords: ai agents, large language models
TripleGrader organizes human review of AI-extracted knowledge
Source published
Shared keywords: human review, large language models
A CT-labeling system gives an AI reviewer fewer decisions to make
Source published
Shared keywords: ai agents, large language models
Can AI Models Help Check Each Other’s Explanations of Legacy Code?
Source published
Shared keywords: human review, large language models
A Rename Is More Than a Find-and-Replace
Source published
Shared keywords: ai agents