An AI’s step-by-step answer is not necessarily an explanation

An AI answer can look more trustworthy when it comes with an explanation. But a convincing account of how an answer was reached is not necessarily an accurate one. A new preprint argues that presenting generated reasoning as transparency risks confusing those two things, particularly in decisions where users need to challenge rather than simply accept a recommendation.

Oliver Lack and colleagues examine chain-of-thought, the intermediate text that some language models generate before producing a final answer. Their concern is not simply that these systems make mistakes. It is that the language of thinking and reconsidering can make a fallible output look like the considered judgment of a knowledgeable mind.

What the paper examines

The paper is a conceptual analysis drawing on existing literature about AI explanations, trust, human judgment and clinical decision-making. It reports no new user experiment, model benchmark or patient-outcome study. Its contribution is an argument about how several familiar features of AI products could combine to encourage inappropriate reliance.

Chain-of-thought gives a model additional intermediate computational space: generated steps can help shape the answer that follows. The authors acknowledge that this can improve accuracy and support evaluation or moderation. They distinguish that technical role from a separate claim—that showing those steps tells a user why the system produced a particular answer.

Users may not even see the complete intermediate text. An interface might show a summary, or brief labels suggesting that the system is planning or evaluating. Such displays are presentations of a process, not necessarily faithful records of the computations responsible for the output. The preprint argues that making the process appear intelligible does not establish that the display accurately explains it.

Human-like presentation is only part of the argument

The authors distinguish between making a system seem human-like and making it seem worth obeying. Anthropomorphism means attributing human qualities, such as understanding or intention, to something nonhuman. A first-person account of working through a problem can supply those cues.

But the paper explicitly does not argue that anthropomorphism alone causes users to defer. People recognise other humans as thinking beings without trusting their judgment on every subject. What matters additionally is perceived competence and authority: the belief that this particular source can reach a better conclusion than the user can.

The proposed risk comes from combining human-like presentation with assumptions about machine precision, objectivity or superior intelligence. Together, the authors argue, those impressions can turn an AI response from something to examine into something that replaces the user’s own assessment.

That distinction also leaves room for useful delegation. Asking a model to organise material is not inherently the same as surrendering judgment. The paper’s concern is reliance that exceeds the system’s demonstrated abilities or obscures what still needs checking.

Why medicine makes the distinction consequential

Clinical decisions illustrate the stakes. A plausible explanation accompanying a recommendation does not establish that the recommendation suits the patient. The authors argue that evidence from well-designed clinical studies, professional expertise and patient values matters more than a fluent account of an individual output.

They also warn against extending success on one task into assumed competence elsewhere. A system’s performance at detecting one condition in scans does not establish broad diagnostic ability across other conditions or patient populations. The paper describes this overgeneralisation as a fallacy of composition: treating competence in one part as evidence of competence across the whole.

Even a faithful explanation would leave another problem unresolved. Accurately describing the process behind an answer does not make the answer correct, nor guarantee that a user will recognise an error.

A warning, not a measured causal result

The analysis does not establish how often reasoning displays cause excessive trust, which users are most susceptible, or which interface designs would reduce the risk. The authors acknowledge that responses to explanations depend on users and tasks. Their broader position on whether generated language constitutes understanding is an argumentative premise, not a finding from a new experiment.

Nevertheless, they recommend discouraging requests for user-facing reasons, limiting reasoning displays in high-stakes settings and teaching their limitations through AI literacy. They leave open the possibility that future research could establish reliable explanatory uses. The central distinction remains: better answers, faithful explanations and appropriately calibrated trust are separate achievements. A step-by-step presentation alone establishes none of them.