When sending an AI query to a smaller model first cannot save money

A smaller AI model can answer easy questions locally and pass harder ones to a larger system. But running the smaller model first creates overhead, and deciding which answers to trust can itself be expensive. A theoretical preprint by Sumit P. Ahuja examines when those costs leave any room for savings—and when no confidence threshold can help.

The price of trying locally first

Tiered language-model inference promises to keep some computation near the user while reserving a larger datacenter model for difficult requests. The practical question is whether the smaller model can handle enough queries reliably to justify the work spent running it.

In A Feasibility Condition for Confidence-Gated Tiered LLM Inference, Sumit P. Ahuja of Main Labs develops a mathematical test for that question. This versioned arXiv preprint is a theoretical position paper, not an experimental demonstration of a new serving system. Its proposed architecture remains future work.

The central result concerns a specific arrangement: every query first runs on a smaller model, a confidence-based decision accepts or rejects its answer, and rejected queries run again on a larger “backstop.” The comparison is sending every query directly to that backstop.

Two requirements must overlap

Ahuja separates the problem into cost and quality. Let κ be the cost of one local pass divided by the cost of an escalation. If the fraction of queries escalated is e, the cascade costs less than the backstop-only baseline precisely when e < 1 − κ.

In plain language, the fraction of requests kept local must exceed the relative overhead of trying locally first.

Quality supplies a second constraint. The router must catch enough wrong local answers without unnecessarily forwarding too many correct ones. Ahuja represents this using sensitivity—the fraction of local errors escalated—and a false-escalation rate—the fraction of correct local answers escalated.

The paper calls the smallest escalation rate that meets the allowed error budget e_q. Its feasibility condition is κ < 1 − e_q. A better-discriminating signal can reduce e_q; more expensive local computation increases κ. Savings are possible only when those quantities leave room for both requirements.

This is conditional, not a claim that cascades generally fail. The analysis assumes the smaller model is less accurate than the backstop and that the permitted accuracy loss is too small to simply use the smaller model alone.

A floor and a ceiling

Ahuja calculates two reference limits: a router with no information about which answers are wrong, and an ideal router that can select only local errors for escalation.

Below the resulting cost-ratio floor, even an uninformative routing signal permits savings within the quality tolerance. At or above the perfect-signal ceiling, no router can meet both requirements under the model. Between them, feasibility depends on the available signal.

The paper illustrates this with assumed—not measured—values: 60% local accuracy, 5% backstop error and two percentage points of tolerated excess error. The floor is approximately 0.057 and the ceiling approximately 0.653. Thus, even ideal discrimination cannot rescue this example if one local pass costs at least about 65% of an escalation.

An uninformative signal, meanwhile, must escalate approximately 94% of requests to meet the quality target. Feasible therefore does not necessarily mean substantial savings or extensive local processing.

Why ownership changes the calculation

Ahuja argues that the limiting factor changes with the deployment’s cost structure. Metered local computation creates overhead that can eliminate savings. When local computation has a sufficiently small marginal cost, that barrier relaxes and signal quality instead determines how much work stays local.

Owned hardware is not literally free, as the paper acknowledges: energy, amortization and opportunity costs remain. Its near-zero-cost regime is an idealization, and the relevant ratio depends on what resource the operator is optimizing.

The distinction also affects self-consistency: generating multiple answers and measuring their agreement. The paper models N samples as multiplying local cost by N. The affordable number therefore depends jointly on sample cost and how much the samples improve discrimination; its conservative reference calculation is not a universal sample limit.

A proposal, with substantial open questions

Ahuja proposes combining three trained-in features: an inexpensive calibrated exit signal, a compressed representation that lets upstream computation resume rather than restart, and placement that keeps local computation local. The paper argues that existing research supplies these ingredients separately, but does not build or evaluate their combination.

The suggested first experiment would check whether a useful early-exit signal exists at the intended model scale. Further comparisons would measure cost–quality frontiers against a tuned cascade, including under distribution shift. Resuming computation from a compressed representation is identified as a particularly uncertain architectural step.

The formal result also has boundaries. It covers two tiers and generate-then-judge routing, not predictive routing before the smaller model runs. It assumes a constant backstop error rate on escalated queries; harder escalated subsets could tighten the constraint. A single cost ratio cannot fully represent independent latency and energy trade-offs.

The paper’s contribution is therefore a feasibility diagnostic under explicit assumptions—not measured proof of cheaper, faster or more private decentralized AI.