More AI compute does not always make better recommendations

When more processing is not more useful

Recommendation systems often work in stages: first retrieve potentially relevant items, then reorder them so the most useful appear near the top. Large language models can perform that second step by reading item descriptions alongside a user’s recent activity. But their additional expense needs to buy a better ranking.

In an arXiv preprint posted on November 5, 2026, Kai Zhao and colleagues at Workday AI Research describe CascadeRec, a system deployed at a financial planning and analysis software provider. Its recommendations concern reports, dashboards, planning models and other workplace content.

The researchers report roughly 10% lower inference costs during a production A/B experiment running for more than a month, with no quality regression observed to date. Their public benchmark experiments offer a more cautionary finding: language-model reranking did not beat the conventional recommendation baseline overall, and sometimes substantially worsened rankings for users with sparse histories.

A second tier, used selectively

CascadeRec runs a lightweight language-model reranker on every request. A cheap routing rule then either accepts that ranking or sends the request to a more expensive second tier.

That tier can sample the same prompt three times and combine the rankings, use three differently framed prompts, or employ a larger model. The purpose is to spend additional processing only where it is likely to help.

The production setting makes that allocation important. Customer organizations differ in both their users’ histories and their request volumes. A small share of customers generates most requests, so uniformly expensive processing can concentrate spending and create latency problems during bursts.

Crucially, the production router does not simply assume that users with less history deserve more computation. Its rule is configured using results measured for particular customer and history-length groups. The paper’s offline policy of escalating sparse-history users was an experiment, not the rule copied into deployment.

Testing the missing-information problem

Because customer data cannot be released, the team also evaluated the approach on MovieLens-1M and Amazon-Electronics. Each dataset contributed 1,000 users, divided into groups whose visible histories were restricted to one, three, five, ten or twenty interactions.

A collaborative-filtering retriever supplied the baseline rankings. Unlike the language-model prompts, this retriever incorporated patterns learned across the wider user population. The researchers tested zero-shot reranking with Mistral-Small-24B and Qwen3.5-27B, without task-specific fine-tuning.

They measured NDCG@10, which rewards systems for placing relevant items near the top of the first ten results. No tested language-model configuration exceeded the collaborative-filtering baseline in aggregate on either dataset.

On Amazon-Electronics, uniform Mistral reranking reduced the score by 29.8% relative to the baseline for users with one visible interaction. Qwen’s corresponding decline was 19.1%. Both differences were statistically significant under the researchers’ paired-bootstrap analysis.

Additional processing was not a reliable remedy. For sparse-history Amazon users, Mistral’s repeated-sampling and multiple-prompt approaches produced still lower scores than its initial reranking. Those results support the authors’ interpretation that missing preference evidence, rather than insufficient reasoning effort, was the main constraint in these tests.

There were exceptions to the negative pattern. Mistral significantly improved the MovieLens ranking score for the group with twenty visible interactions. However, that group had an unusually weak baseline, which the authors say may partly reflect how histories were truncated. They advise interpreting the gain cautiously.

What the production result establishes

The reported 10% saving concerns the entire deployed treatment system, measured through token-level accounting. It does not isolate the router’s contribution from other changes shipped alongside it.

Likewise, “no observed regression” is not a demonstrated quality improvement. The paper provides no production quality effect sizes or confidence intervals because business metrics were still accumulating. Latency impact is described as minimal, but without percentile measurements.

The offline evidence has separate limits. Amazon evaluation used a held-out target alongside sampled negative items, not full-catalog ranking. The history restrictions simulated sparse information rather than entirely unseen users: users were already present in the retriever’s training population. These results therefore do not establish that language models generally fail at recommendation.

Reliability needs its own measurements

The deployment also exposed failures that ranking averages could conceal. In an initial MovieLens setup, Qwen produced unparseable rankings on nearly 97% of requests, often writing essays instead of item lists. Fallback rankings masked the problem in aggregate quality scores. A model-specific output constraint substantially reduced failures.

CascadeRec therefore logs parsing success and falls back to an earlier ranking when a model fails or times out. Its broader lesson is practical rather than universal: measure which groups benefit, verify that the model actually produces usable output, and do not mistake additional computation for additional information.