Why AI Research Ideas Stall After the First Questions

A hidden idea, one question at a time

In the game, an “inquisitor” model sees a research problem drawn from a published paper, but not the paper’s solution. It can ask up to 100 questions, each answered using the full paper. Answers can be yes, no, or unknown. Every 10 questions, the inquisitor writes down its current best account of the paper’s central idea. Three language-model judges score these idea cards against the paper’s solution.

The study, by Muhan Gao and colleagues, applies this setup to 200 ICML 2026 papers and six language models. The goal is not to see whether a model can carry out research: there is no coding, experiment design or laboratory work in the game. Instead, the researchers use the question-and-answer trail to study how models search for an idea and respond to feedback.

The paper’s title calls the format a two-player questioning game, though the evaluation also uses a separate panel to score the idea cards. Its benchmark is built around a specific target: recovering a published paper’s proposed solution from its stated research problem. That makes progress measurable, but it also defines a narrower task than inventing a useful new research direction.

Most progress came early

Across the models, the first 10 questions brought the largest improvement in recovery scores. Questions 11–30 added less, and most models made little progress over the remaining 70 questions. Two models continued to improve slowly, while GPT-5.4 kept climbing more steadily. The researchers also ran a prefix-fork experiment: for each paper, they selected the games that ultimately had the highest and lowest scores, truncated each at question 10, 20, or 30, and continued each prefix three times. The higher-scoring starts generally stayed ahead, suggesting that early search trajectories mattered in this benchmark.

That pattern does not mean later questions are useless in every case. Rather, it shows that many models struggled to recover from an unproductive start. The authors describe two ways searches can go astray. DeepSeek-V4-Flash and Qwen3.6 tended to stay within a small number of lines of inquiry, refining one direction even when it was not promising—what the paper calls over-exploitation. GPT-OSS-120B tended to open many branches without consistently following up on feedback, a pattern called over-exploration. Stronger models showed a more intermediate balance.

Across the recorded games, the researchers found an inverted-U association between the number of broad lines of inquiry and final recovery: games with intermediate breadth tended to score better than games at either extreme. This is an association within the benchmark, not proof that a particular number of branches will improve research in general.

Recognizing is easier than generating

To test where the difficulty lies, the researchers changed what candidate ideas or questions were available. When weaker models could choose among questions proposed by several models, their scores improved substantially compared with asking only their own questions. The gains depended both on having a pool of candidates and on selecting among them; generating several candidates from the same model, or choosing randomly, did not provide the same benefit.

A second intervention gave models a survey of possible approaches to the research problem. When the survey included the paper’s central idea, models—especially those that had scored poorly without it—were more likely to reach that idea. The study reports that models often recognized and used a promising candidate once supplied, even though they had not produced it themselves. This supports the authors’ diagnosis that candidate generation, rather than candidate judgment alone, is an important bottleneck in this setup.

There is an important qualification: the survey was constructed using knowledge of the paper’s solution. It therefore provides information from outside the model’s own search, rather than testing whether the model could find the same candidate unaided. The paper separates this intervention from its baseline game and reports additional checks on how the supplied idea affected the search.

What the results can—and cannot—show

The evaluation has several safeguards, but its scores are not a direct measure of scientific quality. Three language models judge idea cards; human annotators comparing 498 pairs agreed with the judges’ preference or tie in 91.2% of cases. The authors also checked for self-preference because some inquisitor models were used as judges, and report that excluding a model’s own family of judge changed a key intervention’s measured gain by at most 0.08 points. Those checks reduce specific concerns, but do not eliminate the possibility of judging errors—especially when a paper’s idea is ambiguous or has multiple valid formulations.

The answerer is also a model, not a human expert. In a human-annotated sample it agreed with the majority label on 94% of answers, but imperfect answers can still shape the path of a game. The paper’s search trees are themselves produced by a language model, adding another layer of interpretation. And the test set consists of machine-learning papers from one conference; the authors say the approach could be adapted to other fields but did not test that transfer.

The results are reported in an arXiv preprint, which may not have undergone peer review. They offer a detailed way to inspect a specific part of AI research behavior, not evidence that AI systems can independently make scientific discoveries. For automated research tools, the authors suggest that bringing in outside candidate ideas and questions may be worth exploring. Whether that helps with open-ended research, across scientific fields, or in real experiments remains an open question.