AI planners struggle to ask which missing facts would change the plan

An AI assistant asked to schedule vehicles or assign nurses faces two different problems: calculating a good plan and finding out what the calculation needs to include. Records may list a vehicle as available even though it broke down this morning. A staffing request may omit a rule everyone at the workplace already knows.

A new preprint by Yuhan Tang and colleagues examines that second problem. Their benchmark, Ask2Decide, tests whether language-model agents obtain missing facts before committing to a plan—and whether their questions distinguish consequential information from details that sound equally important but do not change the best decision. The authors describe the research as ongoing work.

Making a question’s usefulness measurable

The benchmark contains 43 tasks covering routing, staffing, scheduling and other optimization problems. An agent receives a request and incomplete records. An operator holds additional notes describing current conditions or unwritten rules. The agent can ask questions and use an optimization solver to help construct its plan.

The researchers also maintain a reference model representing the complete problem. This lets them evaluate a submitted plan against the actual conditions rather than merely checking whether it follows the information the agent happened to collect.

To measure a hidden fact’s importance, they calculate a form of “regret”: the unavoidable worst-case loss from choosing a plan without knowing whether that fact applies. They examine its value both when other hidden facts remain unknown and when those other facts are known. That distinction matters because one constraint can become important only after another is discovered.

Some relevant facts have similarly worded controls, called twins. An earlier return deadline for one vehicle might change the best route, while a revised deadline for another leaves it unchanged. Collecting both indiscriminately is therefore different from identifying the fact that matters. The zero-value labels apply to the knowledge conditions checked, not necessarily every possible combination of information.

The interaction rules require questions about specific objects: a broad request for “any other constraints” does not unlock all the notes. Episodes allow at most 15 operator messages and three submitted plans. The first plan is the main benchmark result; later plans measure recovery after feedback.

People do not always explain what went wrong

To make repeated evaluations practical, the researchers developed a simulated operator, CalOp, using human interaction data. Nineteen participants completed 79 sessions across 22 tasks, producing 321 replies.

After an incorrect plan, human replies omitted the correcting fact 43% of the time, compared with 7% for a default language-model operator. A simulator that readily supplies the missing correction can let an agent repair a plan without learning to ask effectively.

CalOp came closer to the observed reply patterns, withholding the correction in 48% of such replies. It was not flawless: factual probes found wrong or invented answers in 1.1% of its responses. Nor does matching selected reply statistics establish that it represents people generally. Participants received hidden notes and assistance, and the reported pooled comparison is not a clean held-out validation.

Better at solving than finding out

The main evaluation tested GPT-6.1 Sol, Gemini 3.8 Flash, DeepSeek V4.1 Flash and GLM-5.3 Flash with CalOp. When agents had to obtain missing information through conversation, their first plans were infeasible on 58–98% of tasks.

The researchers also scored plan quality on a scale where zero means optimal and one means infeasible or a sufficiently large shortfall. Giving agents every hidden fact upfront reduced average first-plan loss to 0.05–0.28, compared with 0.60–0.98 under interaction. These loss values are not failure percentages.

Among the three models that regularly asked questions, operators disclosed 39–52% of relevant facts before the first plan, versus 32–41% of twins. On the larger frontier-A tasks, that preference disappeared. More questions did not necessarily mean better targeting.

Feedback improved final plans, particularly on smaller tasks. But rejection itself reveals that something is wrong, so recovery can conceal weak information gathering.

The findings remain specific to constructed tasks, constrained conversations and the tested models. Main runs used one seed; the two frontier-B tasks lacked proved optima and used bounds and best-known solutions. Larger problems also exposed modeling and solving difficulties even with complete information. Within those limits, the study identifies a distinct weakness: producing a good answer depends on discovering which question needs asking first.