Evolving poisoned search results can mislead AI agents—or make them stop answering

An AI search agent can consult multiple documents and still be manipulated. It might accept fabricated evidence, or encounter a manufactured contradiction and decline to answer. A study by Geyuan Wang and colleagues investigates both outcomes using attacks that change in response to the agent’s behavior.

The paper, The Web Doesn’t Sit Still: Adversarial Self-Evolving Attacks on Search Agents, is available as an arXiv preprint. Its findings concern controlled experiments, not attacks demonstrated against deployed search services. The central contribution is an evaluation framework in which forged-document strategies evolve while defenders receive information about exposed attack patterns.

An evolving population of deception strategies

The method is called Contrastive Rollout Evolutionary Optimization, or CREO. Rather than keeping a fixed collection of attack templates, it maintains a population of natural-language strategies for producing misleading documents.

Each strategy has three components: a description of its deception mechanism, a measured success rate on a fixed evaluation set, and a profile of the question structures on which it succeeds or fails. Named entities are replaced with abstract placeholders when constructing these profiles, helping the system focus on question patterns rather than particular people or places.

A language model examines successful and unsuccessful interaction traces and proposes revised strategies. Candidates are evaluated, and a similarity assessment helps prevent the population from filling with near-duplicates. The experiments cap the pool at ten strategies and run an initialization stage followed by three evolution rounds.

The defender also changes, but not through model retraining. Instead, descriptions of disclosed attack mechanisms and vulnerable question types are added to its system prompt. This distinction matters: the study tests adaptation through supplied defensive knowledge, not an agent independently learning a new security policy.

A controlled substitute for the web

The researchers extend DeepResearchGym with a sandbox containing approximately 197 million FineWeb documents, totaling about 3.8 terabytes. Retrieval settings and result ordering are fixed to reduce retrieval-side variation.

An interceptor inserts a forged document into the agent’s retrieval view without modifying the underlying collection. The document competes with authentic results through a similarity-based ranking process rather than occupying a hardcoded position.

This remains a particular experimental attack surface. The generation process receives the question’s reference answer, and the interceptor prepends the current search query to the forged document before embedding it. These provisions should not be confused with demonstrating that an ordinary malicious webpage would achieve the same exposure or effect on the open web.

The team tests five models—Qwen-Turbo, Qwen3-Max, DeepSeek-V3.2, Kimi-K2.5 and GLM-5.1—on HotpotQA, 2WikiMultihopQA and MuSiQue, sampling 100 training and 100 test questions per dataset.

Being wrong is not the only failure

The evaluation distinguishes two attack outcomes. Direct adoption means the agent uses forged material as valid evidence, even if its final answer happens to be correct. Cognitive paralysis means a conflict introduced by the forgery makes the agent refuse to give a definitive answer.

The paper’s strict accuracy measure therefore requires both a correct answer and the absence of these two outcomes. Its attack success rate measures direct adoption; a separate paralysis rate measures conflict-driven refusal. Their sum is reported as total attack success.

Across the three evolution rounds, average total attack success ranged from 64.2% to 74.6% before the corresponding synchronized defense update, and from 60.5% to 64.9% afterward. The reported averages for the static FactSwap baseline were 20.9% and 18.1%; for AuthChain, they were 55.7% and 37.5%.

These aggregate results support the authors’ claim of stronger disruption in the tested settings. They do not mean direct adoption always increased: substantial disruption came from refusals rather than acceptance of false claims. Because attacks and defenses both change, successive rounds are not a simple comparison against an unchanged opponent. Supplementary fixed-defense experiments provide additional evidence of attack-strategy improvement.

What the results leave open

Paralysis must also be interpreted in light of the agent’s instructions. The supplied system prompt explicitly requires refusal when retrieved documents contain clear contradictions and forbids answering from prior knowledge. Some measured disruption therefore exploits an instructed response to conflicting evidence, rather than demonstrating a general inability to reason through disagreement.

Labels are assigned by a language-model judge using visible interaction traces. The authors report 96–97% agreement with human and additional model annotators on a balanced sample of 100 traces, supporting consistency on that sample without establishing infallibility.

The authors also leave generalization to longer research workflows with more complex tool use unresolved. The practical lesson is narrower than universal agent insecurity: these experiments show why security evaluations should test changing attacks and distinguish contaminated evidence use from an agent being induced to withhold an answer.