EMBERS probes how AI research agents fail when questions and searches change

Deep Research systems do more than generate an answer from a single prompt. They break questions into smaller tasks, search the web, collect evidence and combine their findings. That sequence creates several opportunities for an early misunderstanding to influence everything that follows.

In a November 2026 preprint, Shuochuan Li and colleagues introduce EMBERS, a framework for testing whether these systems remain reliable under changes intended to preserve a question’s meaning and answer. The work targets weaknesses within the research process, rather than attacks that poison webpages or insert misleading external evidence.

Where research goes wrong

The researchers collected 1,932 execution trajectories from DeepDiver, WebSailor and WebExplorer. They manually examined a random sample of 384 trajectories with incorrect answers, tracing the earliest observable error that remained uncorrected and was associated with the final mistake.

Among those examined failures, 65.6% were attributed to subquestion formulation, 24.2% to online search and 10.2% to final reasoning. These percentages describe the distribution within the classified failures—not the probability that each stage will fail. The attribution was observational, rather than a causal finding established through controlled interventions.

The analysis identified nine recurring defect categories. Systems sometimes misinterpreted ambiguous wording, struggled with indirect entity descriptions, dropped necessary context or overlooked important clues. Searches could drift toward popular neighboring topics, respond poorly to query wording or translation, or fail to extract relevant webpage content. Even after retrieving the necessary facts, systems sometimes mishandled counting, ordering or arithmetic.

Turning defects into tests

EMBERS adapts fuzzing: systematically changing inputs to expose failures. Its starting questions must first receive correct answers in three independent runs of the target system. This establishes a more stable passing baseline before mutation begins.

Nine mutation rules then probe the observed weaknesses. Question-level changes include replacing a named entity with a verified indirect description or adding truthful but irrelevant background. Search-level changes include introducing distracting terms, using colloquial phrasing or varying translations. Another rule introduces auxiliary simple operations while aiming to preserve the originally requested output and answer.

For intermediate-search tests, EMBERS preserves the preceding execution history, changes a selected subquestion or query, and resumes the research process. Question-level mutations instead restart the system. Web searches help the language-model mutation operator verify proposed descriptions and factual relationships, although the authors explicitly acknowledge that preservation is not guaranteed.

A semantic answer judge flags cases where a mutated input produces an incorrect answer against the original reference. Correctly answered mutations can also be retained for further exploration, using trajectory length and a diversity proxy based on the originating question and latest mutation rule.

Despite identifying nine defect categories, EMBERS does not directly test all of them. Inaccessible or incompletely parsed webpages fall outside its input-level intervention scope.

More failure indications, not verified unique bugs

The fuzzing evaluation combined the WebSailor and WebExplorer frameworks with both WebSailor-32B and WebExplorer-8B across four datasets, producing 16 settings. DeepDiver was excluded because the researchers lacked executable access needed to inject mutations and resume runs.

Under equal 24-hour budgets within each setting, EMBERS identified 8,340 potentially failure-inducing inputs: 3.21 times the count from DR-Arena, the strongest baseline overall, and 7.87 times that from RobustAlpacaEval. It achieved the highest count in 15 of the 16 settings.

Those figures measure automatically flagged inputs, not distinct underlying defects or independently confirmed failures. Each generated test ran only once, so stochastic model behavior and changing web retrieval could affect the outcomes.

The comparison also has an important qualification: DR-Arena dynamically generates research tasks, and many sampled outputs changed the original information need. An answer mismatch against the original reference can therefore reflect a changed task rather than a genuine robustness failure.

Checking the tests themselves

Human evaluation covered 384 generated inputs from each approach. For EMBERS, evaluators judged that 93.49% preserved the original intent and 88.80% preserved the information necessary to determine the reference answer. They confirmed 88.02% of its automated correctness judgments. RobustAlpacaEval’s paraphrases preserved intent and necessary information at rates above 99%, illustrating a trade-off between targeted changes and fidelity.

The authors also fine-tuned WebSailor-32B using EMBERS-generated OpenSeeker trajectories. On evaluation datasets excluded from training, correctness across all three runs increased from 48.54% to 54.49% on FRAMES and from 60.43% to 69.57% on FRAMES-ZH. These gains do not establish improved resistance to input variations; the paper says further mutation-based evaluation is needed.

EMBERS therefore offers a targeted way to discover candidate weaknesses in the tested research agents. Its results also show why validating the changed question and the answer judgment remains essential before calling any individual case a confirmed failure.