Learning When Human Decisions Matter: Executor-Relative Consultation Value for LLM Agents

Value depends on who uses the answer

The authors distinguish an important decision from a valuable consultation. An agent might already handle a consequential choice correctly, while a seemingly routine question could reveal a private constraint that changes the outcome. Their goal is to predict the improvement in final task performance from asking, rather than simply identify missing information.

They call this value executor-relative: the same answer can help different downstream language models by different amounts. The human does not take over the task. Instead, the agent continues working after receiving information, judgment or authority it lacked.

Learning before deployment

The proposed method, Consultation Value Distillation, or CVD, moves expensive experimentation into offline training. The researchers run the executor repeatedly with different consultation choices, estimate the resulting task utility, and train small neural predictors on those outcomes. The deployed allocator uses information available before consultation, not the hidden answers.

In batch tasks, it scores whole sets of questions. This allows it to represent answers that complement one another, or answers that largely duplicate each other. In sequential tasks, it estimates whether asking now is better than preserving limited capacity for later opportunities. Predictions are averaged across five models to reduce sensitivity to noisy training outcomes.

Once its input features are constructed, allocation needs no new language-model calls or rollout-based value estimation. That is a narrower claim than making the entire workflow cost-free: generating training outcomes, preparing some features and executing the task still require computation.

What the experiments show

The preprint evaluates five batch and three sequential settings spanning six tasks, using Sonnet 5 and GPT-6.1 Sol as executors. Tasks include identifying hidden animals, planning trips and meetings, recommending flights, and controlled business workflows. Learned allocators are tested on executor outcomes independent of their training supervision.

The authors report particularly clear gains when questions must be chosen jointly in the batch Animals task. With three consultations among six candidates, CVD reaches normalized gains of 0.58 and 0.40 for the two executors, compared with 0.25 and 0.09 for their Local Value baseline, which learns each question’s value separately. These differences are reported as statistically significant.

Normalized gain is not an accuracy percentage. It measures improvement between a baseline and an oracle reference. The baseline is random allocation in most settings, but no consultation in the sequential ThinkingBox workflows. Main-table scores are clipped between zero and one.

The results do not establish universal superiority. Local Value remains competitive on several tasks and slightly outperforms CVD in sequential Animals at the largest budget, without a significant difference. For flight recommendation with GPT-6.1 Sol, CVD performs worse than random allocation before clipping. It often leaves consultation capacity unused; requiring it to spend the full three-question budget improves its score.

Efficiency and transfer have boundaries

In the Meeting Planning search experiment, greedy selection followed by single-question swaps matches exhaustive allocation at the four-consultation budget under both executors, using 55 predictor evaluations instead of 386. This demonstrates a cheaper search procedure in the tested setting, not a guarantee for arbitrarily large workflows.

Reusing a predictor across executors also depends on the task. Transfer is nearly lossless on Trip Planning and ThinkingBox-D, but degrades sharply on Animals. The authors interpret this as evidence that learning how a particular executor benefits from answers can matter.

What remains unproven

The study uses simulated or benchmark-defined responses, not live human participants, and modest consultation budgets. Every consultation has the same unit cost. Candidate questions are already specified, and predictors rely on benchmark-specific features; learning useful questions and transferable predictors directly from raw task context remains open.

There is also a sequential-evaluation caveat. ThinkingBox-H largely uses approximate replay that can splice together runs with inconsistent earlier choices or states. A fresh chronological check supports the CVD–Local Value comparison for the tested policies, but covers only the first training run and original test worlds. Training targets and the normalization reference still use approximate replay.

The preprint therefore offers benchmark evidence for allocating human input by predicted downstream benefit, not proof that the method reliably manages attention in real organizations.