WAR-CRAFT audits API recommendations against application requirements

From relevance to documented contribution

Web APIs let applications use functions supplied by external providers. An application combining several services, often called a mashup, needs a suitable set rather than a collection of individually plausible suggestions. Overlapping APIs can add integration work without adding useful functionality, while apparently relevant services may lack documented support for the actual request.

In their October 2026 preprint, Zishuo Xu, Dezhong Yao, and Zhiqiang Xu introduce WAR-CRAFT as a post-recommendation auditing system. It revises a frozen recommender’s output without accessing its parameters, internal scores, or intermediate representations. Its evidence comes from the application request and catalog descriptions documenting API capabilities.

The central distinction is between identifying a possible gap and deciding to fill it. A capability may already be covered, may not require an external service, or may lack a suitable documented provider in the catalog.

How the audit works

WAR-CRAFT begins with a Role Writer, a large language model that evaluates the initial candidates together. It removes APIs whose documented capabilities do not support a requested function. It can also remove a supported API when another retained candidate fully subsumes its relevant contribution, provided no distinction required by the request is lost. Partial overlap alone is insufficient.

Each retained API receives a short functional role explaining its contribution. Canonical API names are withheld during this judgment to focus attention on documented capabilities.

Next, a Joint Requirement Auditor constructs a checklist of external-service requirements and assesses coverage in the same model call. Requirements must be grounded in the request; coverage must be supported by API descriptions. The generated roles provide context but cannot establish undocumented capabilities. Checklist entries include evidence from the request and, where covered, supporting description excerpts.

For each missing requirement, a fine-tuned Resolver chooses an eligible catalog API or abstains. Its output choices are constrained to catalog-specific action tokens and a no-add option, preventing proposals outside that action space. Initial candidates, including deleted APIs, cannot be proposed again. Multiple proposals for the same API are deduplicated.

A separate Commit Verifier then checks each proposed API’s description against the requirements that prompted its selection. An API is accepted if the description supports at least one originating query—not necessarily all of them. Accepted additions receive functional roles, and the system records its decisions and evidence in an audit trace.

What the experiments found

The authors evaluated seven upstream recommenders using a ProgrammableWeb corpus containing 8,217 mashups and 1,647 APIs, with 1,643 mashups reserved for testing. The same refinement configuration was used across all seven sources.

Mean SetF1 increased from 0.5580 to 0.6444. This metric balances precision and recall against the APIs recorded for each mashup; it is not a direct measure of whether an application works. The reported improvement was 8.64 percentage points, with a 95% paired-bootstrap confidence interval of 8.01–9.28 points. Every upstream source improved, although gains were larger for fixed-cutoff ranking systems than for generative recommenders.

Average functional redundancy fell from 0.1265 to 0.0654, while average output size declined from 1.86 to 1.59 APIs. Mean functional coverage showed no statistically significant change. Coverage decreased for two sources, and redundancy increased for the GPT source, so the averages do not describe every recommender’s outcome.

WAR-CRAFT also exceeded the strongest independent refinement baseline’s mean SetF1 by 1.92 percentage points. Comparisons with equal-length prefixes indicated that reducing set size alone did not explain the improvement.

Why restraint matters

The Resolver abstained on 69.7% of requirement-level decisions. In a separate control, always adding the top retrieved API for each missing query produced lower SetF1 than filtering alone. These results support the authors’ argument that addition needs its own decision, rather than following automatically from gap detection.

The final verification stage rejected 4.6% of deduplicated proposals but barely changed mean SetF1. Its demonstrated contribution was primarily documentation-based screening and role generation, not a substantial accuracy gain.

Limits of the evidence

WAR-CRAFT performs one refinement pass without re-auditing the final set. Requirements left as composite queries may remain unresolved because the Resolver selects at most one API per query.

The evaluation also depends on historical API sets and coarse functional labels, which cannot fully recognize valid alternatives. Human audits involved the authors, and the semantic stages used proprietary GPT-5.6 Sol. The authors acknowledge that possible exposure to ProgrammableWeb during pretraining cannot be ruled out. The findings therefore support improved benchmark recommendations, not a guarantee of complete requirements coverage or successful live integration.