Can an AI disaster report show its evidence?

From image pairs to an evidence trail

The central question is whether an AI system can generate a broad disaster report while making its claims traceable to evidence in imagery. DisasterAgent takes a pair of optical satellite images, one from before and one from after an event, together with a natural-language request. Instead of asking a vision-language model (VLM) to write directly from the images, it first runs perception and evidence steps.

Specialized vision tools identify scene elements and assess changes, including building damage, road impacts and land-cover changes. The system organizes these outputs in a Spatiotemporal Perception Graph (STPG): a structured record of regions in each image and links describing how they appear to change between the two dates. A region might, for example, be matched across images and marked as expanded, diminished or disappeared.

The system then assembles several kinds of supporting material. It creates individual claims about changes and checks each against the original image pair, retaining claims that pass a confidence threshold. It also draws annotations on the images to show where mapped features or damage are located. A retrieval step finds descriptions and restoration advice from historical cases in the DisasterM3 training data. The VLM then generates a report with inline citations pointing back to claims and mapped regions. Restoration advice is generated in a second stage, using the report and retrieved historical advice rather than the raw images alone.

This design aims to let a reviewer trace a statement back through the system to relevant image regions. A citation, however, does not guarantee correctness. Given the described pipeline, errors in perception, region matching or claim verification could affect the evidence chain; this is a limitation implied by the method, not a reported measurement of error in each component.

What the benchmark tests found

The authors tested DisasterAgent on two benchmarks. RSCC evaluates captions of changes between image pairs; DisasterM3 covers a wider set of disaster-assessment tasks, including report writing and restoration advice. On DisasterM3, the authors report a 13.5% improvement in the average disaster-report-generation score compared with the base model. On RSCC, DisasterAgent’s ST5-SCS score was 67.18%, compared with 51.84% for the base Qwen3.5-27B model. The paper describes this as a 15.34% improvement; the difference between the displayed scores is 15.34 percentage points. ST5-SCS measures semantic similarity between a generated caption and a reference caption. These are benchmark results, not measured improvements in real-world response outcomes.

The comparisons also suggest that structure may add value beyond simply giving an AI more tools. In an ablation study of the tested configurations, adding a specialist toolkit improved the overall report score, graph-based organization raised it further, and the complete evidence-augmentation setup achieved the highest score among those configurations. Removing historical-case retrieval also reduced scores, including for disaster descriptions as well as advice. These results indicate that the components contributed in the experiments; they do not establish that each will help in every setting.

The authors report that citations to semantic-change claims appeared in 94.86% of evaluated DisasterM3 samples and 85.32% of RSCC samples. Citations were therefore not present for every sample. These rates do not by themselves measure whether a human responder can verify claims efficiently or whether citations prevent consequential errors.

What remains uncertain

The evaluation uses existing image-pair benchmarks and compares generated text with reference captions or scores it using a language model as judge. Such tests allow system comparisons, but they cannot show that reports improve decisions during an emergency. Nor do the scores establish reliability across all image conditions, locations or disaster types. The paper focuses on optical imagery; its authors identify adding other data sources, such as radar imagery, and extending analysis to more than two time points as future work.

The graph relies on outputs from specialist models, and the authors note that upstream perception errors can affect later claims. Checking claims with another VLM may filter some mistakes, but it is not the same as an independent human audit. Historical examples may also guide a report without being a close match for the event at hand. These uncertainties matter when recommendations could influence response priorities.

The supplied source is the first version of an arXiv paper. Its findings should be treated as research results, not as proof that the system is ready for operational emergency use. The work’s central contribution is a design idea: combine structured image analysis with report generation and explicit links from claims to visual evidence. Whether that approach remains dependable in real deployments will require further testing, including evaluation by human experts and response teams.