AI satellite analysts struggle to separate land change from sensor change

A test built around satellite archives

Haoran Zhang and colleagues introduce EarthRecord, a benchmark covering 56 sites and 238 questions about forests, agriculture, water, cities, fire, snow and coasts. The archive combines Landsat, MODIS, VIIRS and Sentinel-2 observations spanning 1984–2026, although coverage varies by satellite and location.

Unlike tests based on selected, ready-to-view images, EarthRecord supplies satellite files as distributed by their providers. Agents must apply scale factors, screen clouds, handle missing observations and reconcile measurements from different instruments. Some duplicate and heavily obscured scenes were removed beforehand, but other archive problems were retained.

The researchers evaluated 12 multimodal AI agents: systems that can work with both text and images and use software tools. Each received an offline, CPU-only environment with Python, geospatial libraries, a shell and an image viewer. Runs had a one-hour working-time limit.

Questions ranged from calculating vegetation trends to identifying forest loss or explaining shoreline change. Twelve questions deliberately lacked enough observations for a supported answer. Agents submitted a conclusion, confidence estimate and account of their methods.

Reference answers came from several sources: agreement among calculation methods, global mapping products, and human-reviewed visual interpretations. Some mapping products were withheld from the agents, requiring them to estimate the relevant changes themselves.

Strong calculations, weaker mapping

The main score required a correct conclusion in all three independent runs of a question. This measures repeatability as well as accuracy; it is not ordinary single-attempt accuracy.

On the 226 questions considered answerable, the four leading agents scored between 81.0% and 85.4%. Their uncertainty intervals overlapped, so these results do not establish a clear ordering among them.

A sharper distinction appeared between types of analysis. Across the six leading models, scores averaged 92.0% on 98 questions calculated from variables in the record, compared with 61.1% on 78 questions requiring mapping from imagery.

Some systems declined mapping questions after only a few tool calls, usually without viewing an image. Others completed an analysis but made consequential judgments incorrectly—for example, interpreting harvested fields as burned land. Different runs of the same model could also choose different definitions or statistical methods and reach conflicting conclusions.

Changing the measurements, not the land

Two controls examined what supported the answers.

First, a closed-book condition removed the satellite archive while retaining the question and site coordinates. Every model improved on calculation-based questions when given the record. Improvements were less consistent on questions about what happened on the ground: four models scored worse with the archive than without it.

Second, the researchers created 22 paired stress tests by altering one satellite’s data while keeping the underlying landscape and reference answer unchanged. Most introduced shifts from 2018 onward; another substituted observations from a different Landsat satellite. Each alteration could be detected by comparison with an overlapping satellite record.

Among 200 model–question cases answered correctly in all three original runs, 107, or 53.5%, lost that perfect score after alteration. In a separate run-level analysis, 91% of answers whose stated direction changed shifted in the direction encouraged by the altered measurements.

For comparison, placebo tests that randomly removed 5% of clear scenes disrupted 8 of 66 previously correct cases, about 12%.

Only 3.5% of the wrong runs in disrupted instrument-pair cases attributed the break to the instrument. Some noticed disagreement between satellites but still interpreted the result as land change or left its cause unresolved.

What the results can—and cannot—show

These findings come from a preprint and should be treated as preliminary. The benchmark covers selected sites, not a representative sample of every landscape or monitoring task. Its reference answers are operational standards based on products, methods and interpretation, rather than universal ground truth. Human reviewers checked visual references initially drafted by AI.

The evaluation also used CPU-only environments without pretrained vision models for mapping. Its instrument tests introduced clear breaks, not the full range of subtle drift encountered in practice. Scores therefore describe these agents under this particular setup, not every possible AI-assisted workflow.

One further qualification concerns benchmark development: after the first model’s pilot run, the authors corrected references and removed 11 unstable questions. That model was rescored; the others used the frozen question bank.

The central finding is a distinction between using evidence and validating it. Agents often calculated successfully from satellite measurements. They were less reliable at checking whether those measurements supported a conclusion about the land itself.