Reading the warning signs: an explainable way to stop failing web agents early

Predicting failure, not proving inevitability

In their preprint, Interpretable Early Termination of Web Navigation Agents via Closed Sequential Pattern Mining, San Jose State University researchers Sergio Talavera and Magdalini Eirinaki propose a readable alternative to opaque failure scores. Their detector looks for recurring sequences associated with unsuccessful runs and reports which patterns triggered a stop.

The distinction matters: a pattern predicts failure from past examples; it does not prove that a particular run cannot recover. The study therefore evaluates both token savings and the risk of interrupting agents that would otherwise succeed.

Turning execution logs into a small pattern library

The researchers collected 1,544 execution traces across 10 medium-difficulty MiniWoB++ tasks using Llama-3.2-3B, Qwen-2.5-7B and Mistral-7B-Instruct-v0.3. These included 621 successes, 290 natural failures, 315 timeouts and 318 errors. Agents used ReAct prompting, which interleaves reasoning and actions, with a 30-step budget. Task outcomes came from the environment’s completion signal.

Each step became a symbolic description combining its action, how the target element was selected, the observed outcome and a reasoning-intent label. Those intent labels, such as verification or retrying, were extracted through fixed keyword and regular-expression rules rather than a demonstrated understanding of the agent’s reasoning.

The method examines prefixes—the first three, five, eight or ten steps—and applies BIDE+, a sequential-pattern-mining algorithm. A pattern is “closed” when no longer pattern appears in exactly the same set of traces. This reduces redundant entries while preserving information about how often patterns occur.

At a minimum support of 5%, mining produced 33 patterns. Removing patterns whose failure precision was below 50% left 32. During execution, the detector searches for these patterns as ordered subsequences; their steps need not be consecutive. Matched patterns’ precision scores contribute to a coverage score, and a threshold determines whether to stop. The explanation includes the matched patterns, their positions and a failure category.

Manual inspection organized the library into recovery, validation, navigation and context failures. Examples include repeated verification clicks, extended click sequences without other actions, and typing interrupted by an unparseable action. The authors associate these categories with architectural repair recommendations, but do not test whether those recommendations improve agents.

Separating behavior from shortcuts

A pilot exposed a major evaluation risk: crashes before any action and successes completed in one step can make trace length an easy shortcut. A classifier may appear effective without recognizing meaningful behavior.

The main medium-task corpus avoids single-step successes by design. The researchers also compare configurations that progressively remove errors and then timeouts. A length-only baseline tests whether behavioral methods outperform that shortcut. Data were split 60/20/20 into training, validation and test sets at a fixed seed, with decision thresholds tuned on validation data.

On the balanced Exclude-errors configuration, which retains timeouts as failures, BIDE Coverage achieved the highest three-step F1 among eight methods: 0.717. F1 summarizes the balance between precision and recall. The frequency-vector baseline was close at 0.712. At ten steps, the bidirectional LSTM performed better: 0.765 versus BIDE’s 0.612.

When both errors and timeouts were removed, BIDE led the comparison from five steps onward, reaching F1 of 0.683 at five steps against 0.667 for the length-only control. These are modest margins, not evidence of universal superiority.

The library matched at least one pattern in 88% of test-set failures within three steps. That measures pattern coverage, not the fraction of failures actually stopped by the recommended policy.

What the stopping policy saved—and sacrificed

The savings analysis used a separate, natural-prevalence Exclude-errors test set containing 245 traces: 121 failures and 124 successes. At five steps and a threshold of 0.15, the policy stopped 50 runs. Of those, 46 were failures and four were successes.

That corresponds to 92% precision, 38% failure recall and a 3.2% interruption rate among successful runs. Reported overall token savings were 26.3%. Lower thresholds saved more tokens but interrupted more successful runs; stricter thresholds reduced both savings and false stops.

Limits on deployment claims

Performance varied substantially by backbone: three-step F1 on Exclude-errors ranged from 0.526 for Llama to 0.913 for Qwen. Additional WebArena shopping traces showed partial, model-dependent pattern coverage, not established transfer of the MiniWoB++ savings result.

Interpretability is also a structural claim here: the system returns readable evidence, but no user study establishes its usefulness to operators. The taxonomy lacks a documented coding protocol or inter-rater agreement. Results come from one fixed-seed split without multi-seed variation or significance tests.

The preprint supports an inspectable trade-off between early stopping and continued execution—not a guarantee that flagged agents are beyond recovery.