Teaching AI when to ask another question—and when to stop

“How much does that one cost?” is easy to answer only if an assistant knows what “that one” refers to. Otherwise, it needs another question—and a way to recognize when further questioning would waste time.

In When to Ask, When to Answer: Strategic Clarification for Ambiguous Reference Resolution in Dialogue, Colorado State University researchers Ibrahim Khebour and Nikhil Krishnaswamy investigate this stopping decision. Their paper, submitted to arXiv on November 3, 2026, is a preprint; the supplied landing page does not report a peer-reviewed publication venue.

The researchers propose BACE, short for Belief-Aware Critic with Epistemic tracking. Rather than retraining a language model to both write responses and choose its next move, BACE freezes a supervised-fine-tuned response generator and trains a separate chooser. In this controlled setting, the approach performs better than two reinforcement-learning alternatives at deciding how long to keep clarifying. Its overall improvement over supervised fine-tuning alone is much smaller.

A reference game with a price for questions

The team builds its environment from SIMMC 2.1, a shopping-dialogue dataset. Each episode provides dialogue history, a user request and a list of candidate objects. The assistant receives non-visual attributes such as brand, price and size, along with positions and spatial relations. Visual attributes such as color are withheld.

At each turn, the assistant can ask about one of eleven permitted attributes or answer with an object identifier. A scripted user replies according to the designated target object. These replies allow candidates to be ruled out.

Correct answers earn one reward point; wrong answers lose one. Each question costs 0.1, with a budget of three questions. The objective therefore discourages both premature guesses and unnecessary exchanges.

An oracle procedure assigns each episode a reference question count, which the authors call its gold depth. Evaluation measures not just correctness, but whether a system answers after that many questions. Matching the count does not itself guarantee a correct answer, so the paper also reports “hit-and-correct.”

Keeping track of what remains possible

BACE reconstructs a belief set: the candidates still compatible with the observed clarification replies. This is not a direct feed of the environment’s internal survivor set; it is maintained from the information available to the system.

At each turn, the frozen generator produces a candidate question and a candidate answer. The chooser scores them using language-model representations and explicit features, including how many candidates remain, how much question budget is left and how informative the proposed question is expected to be. Training rewards task success minus question costs, rather than information gain itself.

All main methods use Qwen2.5-7B-Instruct. Their shared supervised starting checkpoint learns from oracle demonstrations using real and synthetic training episodes. Subsequent reinforcement learning uses real episodes, and evaluation uses real development and devtest episodes.

Stronger timing, modest overall gains

On the 2,483-episode devtest split, BACE achieves mean reward of 0.940, compared with 0.938 for supervised fine-tuning, 0.926 for trajectory-level GRPO and 0.918 for the authors’ ArCHer adaptation. Reinforcement-learning results are averaged over five seeds. BACE’s accuracy is 99.2%, versus 99.0% for supervised fine-tuning.

The differences become more revealing in episodes needing several questions. In the two-question group, BACE answers at the reference count and correctly in 85.3% of episodes, compared with 66.0% for GRPO and 62.3% for ArCHer. Supervised fine-tuning remains slightly ahead at 85.9%.

In the hardest group, BACE reaches 56.2% hit-and-correct, versus 47.9% for supervised fine-tuning, 25.0% for GRPO and 20.0% for ArCHer. That group contains only 48 episodes and includes cases requiring more than three questions or impossible to resolve with the available attributes.

What the experiment does not establish

The task makes ambiguity unusually explicit: an unambiguous episode lists only one candidate. Real assistants would first need to identify the possible objects themselves. The study also uses one dataset, one backbone and exact scripted replies—not users who misunderstand questions or answer vaguely.

Target labels introduce another qualification. Only 49% of ambiguous devtest episodes obtain their target from a later dialogue turn naming one object. The remainder use fallback selection, so the scripted target may not match the original user’s intention.

BACE is not cheaper merely because it trains fewer parameters: its reported additional training takes 18.2–19.0 GPU hours per run, versus 3.4–3.5 for GRPO and 5.2 for ArCHer. Shorter BACE runs also show instability.

The supported lesson is specific: explicit candidate tracking and a separately trained chooser can improve clarification timing in this reference game. Whether that advantage survives visual perception and less predictable human dialogue remains untested.