When Should an AI Agent Look Outside the Graph?

Knowledge graphs store information as connected facts: a sensor measures a quantity, a piece of equipment occupies a location, or two components connect. Rules can flag when a graph is incomplete or inconsistent. But passing those rules does not necessarily mean the graph reflects reality. Several different edits may satisfy the same rule, and the smallest edit may not be the right one.

Consider a building graph that lacks a connection between an air-handling unit and a zone. A single added link might make the graph valid. But an equipment schedule could show that the actual air path passes through an intermediate device, requiring several additional nodes and links. The question explored in this study is whether an AI repair agent should be allowed to consult documents like that schedule—and how to decide when the extra search is useful.

A repair agent, with search sometimes switched off

The researchers built a system with an AI repair agent that explores and edits a graph, checks its changes against schema rules, and can optionally ask a separate “researcher” agent to search documents. A router decides whether the repair agent gets access to that researcher for each group of related violations. When access is available, the repair agent still chooses whether to ask a question; access does not force a search.

To evaluate repair quality, the team needed cases with reference repairs. They started from valid graphs supplied by the dataset authors, deliberately introduced compatible violations, and recorded the edits needed to reverse them. The study treats those supplied graphs as factually correct; their correctness was assumed, not independently verified. The researchers tested on two building-automation datasets, Pritoni and IBAL, each paired with a schema and a document collection. The system used Gemma-4-31B for its language-model calls.

The evaluation tracked three things: how closely proposed edits matched the reference repair, whether the final graph satisfied the schema, and how many tokens the repair process used. The researchers first compared repairs with and without researcher access, then trained a router using outcomes from generated cases. The router used the predicates involved in a violation cluster to predict repair quality with access and without it, and enabled access when its prediction favored doing so.

Search helped in some cases, hurt in others

The central finding is that external retrieval did not have one consistent effect. Its usefulness varied with the kind of repair and the particular deployment. In the IBAL tests, access produced a strong median repair-F1 improvement for violations in the study’s “evidential” group—relationships commonly documented in engineering records. The improvement was smaller for the “terminological” group. For “representational” violations, access was associated with worse median repair F1. In Pritoni, the median changes were positive across all three groups, but generally smaller.

One example illustrates how a search result can misdirect an agent. While trying to repair a missing connection-point link, the agent with researcher access received a description of the component’s air connections. It then searched the graph using terms from that description but failed to find the existing connection-point node. The agent created a new node instead. The authors present this as one observed trajectory, not proof that the same failure will occur generally.

The evaluation comprised six suites: three predicate mixtures—evidential, terminological, and representational—for each of the two datasets. Across these suites, the learned router used fewer repair-time tokens than a policy that always made research available. It achieved higher repair F1 than a policy that never made research available in five of the six suites. Compared with always-on access, it had higher repair F1 in four suites; the differences in the two evidential suites were not statistically significant. These results do not mean the router was best on every measure in every suite. In the IBAL representational suite, the router improved on always-on access but remained less accurate and more costly than the no-search policy.

Schema validity was high across conditions: the paper reports that fewer than one percent of the 2,400 policy runs failed to produce a valid graph. That measure is distinct from repair F1. A graph can satisfy the rules yet fail to match the reference repair—precisely the distinction motivating the study.

What the evidence cannot yet show

The router was trained separately for each dataset, using generated cases and the dataset’s reference graph, then evaluated on separate generated cases. This is evidence for a deployment-specific approach, not a demonstration that one router transfers to unrelated domains. The tests cover only two building-automation deployments using the same schema family. The authors say broader evaluation needs datasets that include a schema, a reference graph, and aligned supporting documents.

There are further uncertainties. The study treats the document collection and the researcher agent as one retrieval pipeline, so it cannot tell whether a missing fact was absent from the documents or present but missed or misinterpreted during search. The router is trained offline and does not adapt automatically when documents or retrieval systems change. Its training also depends on a trusted reference graph and incurs preparation costs beyond the repair-time token counts reported in the comparisons. The authors note that the current decision rule considers predicted repair quality, not whether the expected improvement justifies the token cost or how uncertain the prediction is.

The paper is an arXiv preprint; the supplied material does not establish that it has undergone peer review. Its results make a useful case for selective rather than automatic access to external evidence, while leaving open how well the method will work beyond these particular building datasets and controlled repair cases.