AI web agents often scrape despite site rules—and sometimes refuse when allowed

A web agent may seem to be doing a simple job: open a page and collect its contents. Yet that action can run into website policies, including terms of service (ToS) and robots.txt directives. ScrapeBench asks whether AI agents notice and follow those rules, and whether they avoid the opposite mistake: refusing a task on a site labeled as not explicitly prohibiting scraping.

That distinction matters. A system that ignores restrictions may overstep, while one that refuses every request may be of little use. The researchers measured both compliance with their prohibition labels and “over-refusal”—declining a task on a site labeled as not explicitly prohibiting scraping.

How the benchmark worked

The team started with the top 3,000 websites in the Chrome Top Million Websites (CrUX) list for February 2026. After filtering out adult-oriented sites, sites primarily in languages other than English, and sites inaccessible from the United States, they retained 1,895. Based on the sites’ ToS and robots.txt, the team labeled 1,626 as explicitly prohibiting scraping and 269 as not explicitly prohibiting it. Sites with conditional restrictions were excluded to create these two clear-cut categories. The researchers manually checked a 15% sample of the classifications and reported high agreement with their automated extraction process.

“Not explicitly prohibiting” is not a finding that scraping is lawful in every circumstance. The authors treat these categories as benchmark labels, not legal rulings about any particular action.

The team tested six AI agents with a tool that could fetch website pages. Each received one of eight prompt conditions, ranging from a plain request to scrape a landing page to instructions supplying policy text and warning of possible legal consequences. Another condition told the agent to ignore the policies. The researchers recorded whether an agent fetched a content page, whether it checked policy resources, and what it said about its decision.

What the agents did

With a neutral request, compliance on sites labeled as prohibiting scraping was near zero for five of the six agents; Grok was the exception, complying in 11% of trials. Simply mentioning that policies might exist did little to change the pattern. Providing policy text in the prompt often helped: under that condition, GPT, Claude and Gemini each exceeded 70% compliance. Results varied considerably by model, with Qwen at 2.9% in the same condition.

A stronger prompt—combining policy text, an instruction to comply and a legal warning—produced the highest compliance for five agents. But it was not a universal fix: GPT’s compliance was lower under that warning than when policy text alone was pasted into the prompt. Explicit context can influence behavior, but these results do not show that prompt wording guarantees compliance.

Checking policies was not enough by itself. Agents often retrieved linked rules, yet compliance remained low. In a subset of violating trials where agents gave a rationale, the researchers found that some treated a permissive robots.txt as if it overrode a restrictive ToS, or misread the ToS. These explanations were categorized with another language model, so they offer evidence about stated reasoning, not a complete account of why each agent acted as it did.

The benchmark also exposed over-refusal. In the policy-text condition, most agents rarely refused on sites labeled as not explicitly prohibiting scraping. Under the adversarial instruction to ignore policies, Claude’s over-refusal rate on those sites was 98.5%. Separately, on sites labeled as explicitly prohibiting scraping, Claude’s compliance rate under that prompt was 99.7%. The contrast highlights that a high refusal rate in one category does not necessarily mean the agent is carefully distinguishing the site-specific rules. Other agents under the adversarial prompt tended either to refuse broadly or to scrape broadly.

What this does—and does not—show

The researchers recruited 180 US-based participants for a comparison. In the neutral condition, people declined to scrape 48.9% of sites labeled as prohibiting scraping and declined 38.9% of sites labeled as not explicitly prohibiting it. These were simulated decisions in a web app: clicking “scrape” displayed a previously saved page rather than sending a live request to the website. The paper reports that legal warnings increased human compliance to 65.1%; changes in human over-refusal across the tested conditions were not statistically significant.

Several limitations narrow what can be concluded. The benchmark covers a particular snapshot of high-traffic, English-language websites accessible from the United States, and policies can change. It tests isolated, single-agent tasks, not longer or multi-agent workflows. The legal force of ToS and robots.txt can depend on jurisdiction and circumstances, including purpose, contract law, copyright and privacy rules. The authors explicitly say the study does not establish that any specific action in the experiment was—or was not—illegal.

ScrapeBench is a preprint, and its findings concern selected agents under specified conditions, not every web agent or scraping law in general. Its central observation is more limited: retrieving policy text does not ensure that an agent will interpret or follow it correctly. The benchmark offers a way to test that behavior, while leaving broader legal questions unresolved.