When AI learns new facts, its confidence may lag behind

Learning and judging are different tasks

A language model can become better at answering some questions while becoming worse at recognizing which answers it is likely to get right. That is the central problem examined by Zichuan Fu and colleagues in Feeling of Knowing in Large Language Models, an arXiv preprint whose landing page reports acceptance to NeurIPS 2026.

The researchers use “feeling of knowing,” or FoK, as a functional label for an answerability judgment: can this model answer this question correctly? They explicitly do not mean that the model has feelings, consciousness or human-like introspection.

The distinction matters for systems deciding whether to answer, request outside information or pass a question to another model. An outdated judgment could encourage an unsupported answer—or reject a question the updated model could handle.

What changed after training

The authors first investigated what happens when factual-answer training and judgment training are performed separately. Across seven tested models, conventional factual fine-tuning increased answer accuracy, but the ability to distinguish correct from incorrect answers declined in six.

The paper measures that distinction using AUROC, a score describing how well answerability judgments rank correct answers above incorrect ones. A score of 0.5 indicates chance-level discrimination; higher is better. It is not the percentage of answers that are correct, nor does it establish that a stated confidence level matches the actual success rate.

The main experiment used Qwen2.5-7B-Instruct, with 5,000 TriviaQA questions for updating and 1,000 held-out questions for evaluation. Conventional fine-tuning improved performance on the update questions but reduced held-out accuracy from 67.2% to 54.6%. Held-out FoK AUROC also fell, from 0.688 to 0.519.

This was therefore not simply successful learning accompanied by misplaced confidence. In that experiment, conventional training also generalized poorly to unseen questions.

A separate update for answerability

The proposed method separates changes to factual answering from changes to answerability judgment, using two trainable adapters attached to the base model. Training proceeds in three stages.

First, the model generates answers to training questions. When an answer is wrong, the researchers replace the answer-relevant span with a correct answer. Training targets that span rather than the entire response. This approach, called on-policy supervised fine-tuning, aims to learn facts while limiting disruption to existing behavior.

Second, a judgment adapter learns across checkpoints saved during that factual training. The same question can receive different correctness labels as the model changes. The goal is to make judgment track the evolving model rather than memorize a fixed list of answerable questions.

Third, policy optimization trains a combined response: generate a short reasoning trace, judge answerability, then answer or explain why the model cannot answer. Rewards penalize incorrect judgments and answers, including refusing questions labeled answerable.

In the main reported run, the full method reached 69.2% held-out accuracy and 0.809 AUROC. The intermediate judgment-training stage supplied much of the improvement in discrimination; the final stage further improved judgment while slightly reducing answer accuracy.

Better discrimination, not certainty

Errors remained substantial. At the paper’s decision threshold, the full method predicted that it could answer 43.2% of the questions it actually got wrong. Conversely, it predicted that it could not answer 18.4% of questions it would answer correctly. These are conditional error rates, not percentages of all questions.

The comparison also does not establish universal superiority over existing confidence methods. Applied to the base model, the EliCal baseline achieved 0.810 AUROC—essentially matching the full method’s 0.809 in the main table, although answering accuracy differed.

Additional experiments tested transfer to other factual-question datasets and three sequential updates. Retraining judgment each round maintained AUROC near 0.80, while a frozen judgment adapter deteriorated. However, held-out answer accuracy remained below the original model’s level. Keeping judgment synchronized did not demonstrate cumulative improvement in factual answering.

Where the evidence stops

The authors report training-seed variation, but the final optimization runs shared one starting checkpoint. Those results do not measure variability across independently trained full pipelines. Answer correctness in the main TriviaQA evaluation also relied on another language model to judge equivalent answer wording.

Context remained a vulnerability: incorrect but related sentences increased answerability scores while slightly decreasing accuracy in a supplementary experiment. More confident judgment was not necessarily more truthful judgment.

The full framework was tested mainly on short-answer factual questions with gradient-based updates. Its effectiveness on long update sequences, code generation, mathematical proofs or arbitrary editing methods remains unestablished. The practical finding is narrower: in these experiments, updating what a model can answer did not reliably update its judgment of that ability, and explicitly training that judgment helped.