When AI gets the answer right, what should people understand?

An AI system might produce an acceptable result and pass the relevant checks while leaving people unable to explain its implications. Would that matter if further human checking would not improve the result? Liming Zhu, Qinghua Lu and Ming Ding of CSIRO argue that it can: people may still want to choose different goals, deliberate about public decisions or understand a discovery out of curiosity.

Their preprint proposes understanding-first AI engineering. The idea is to specify human understanding as an intended outcome early enough to influence system design, rather than assume that supplying an explanation achieves it. “First” does not mean people must understand everything before an AI acts, or that learning overrides safety, privacy and cost.

Understanding for whom, and why?

The authors distinguish four overlapping reasons to support understanding. Reflective agency means working out what to pursue and why preferences might change. Effective action means understanding enough to apply a result, diagnose a fault or recognise a limitation. Collective governance concerns shared decisions, authority, trade-offs and accountability. Epistemic value means valuing understanding for its own sake.

These purposes imply different requirements. An operator might need to recognise when an unfamiliar condition warrants escalation. A community representative might need to distinguish forecast uncertainty from a disagreement about priorities. Neither necessarily needs the same technical knowledge as a scientist.

The proposal therefore asks designers to identify the recipient, subject, required depth, permitted tools and resource budget. Where assessment is appropriate, it should examine relevant reasoning—not merely confidence, a fluent answer or completion of an exercise. Voluntary exploration motivated by curiosity need not involve formal assessment.

More than an explanation interface

The proposed architecture connects production, verification, use, and learning and exploration through shared records. Those records preserve evidence, assumptions, attempted approaches, limitations and revisions. They must distinguish established findings from plausible explanations and exploratory proposals.

These are responsibilities, not a requirement to build five separate services. Existing repositories, teaching tools and team practices might already provide them. The important question is whether someone owns the necessary investigation, review and follow-up.

The authors also propose five candidate design patterns. A parallel learning workflow lets people practise or reconstruct results without delaying unrelated operational work. A separate explanatory model offers an accessible representation alongside a more complex production model. Branch and compare goals keeps exploration of alternative priorities separate from authorised commitments. Audience-specific views connect differently detailed accounts to shared evidence. Tested handover asks a receiving team to demonstrate that it can locate expertise, apply relevant knowledge and identify unresolved questions.

Each carries costs. Learning materials become outdated; simplified models can mislead; alternative scenarios can frame preferences; tailored accounts can conceal disagreements. A simpler arrangement is preferable when it meets the same requirements.

A particularly important boundary concerns explanatory models: matching an AI system’s predictions does not establish how a real phenomenon works. Claims about a physical mechanism need independent supporting evidence. Likewise, establishing an explanation’s accuracy and establishing a person’s understanding are separate tasks.

Two hypothetical demonstrations

The paper develops its framework through selective literature synthesis, experience from previous projects and design reasoning. Its evaluation consists of author-conducted analysis of two hypothetical scenarios, not experiments with deployed systems or participants.

In the first, an energy-storage material already meets accepted performance limits, but two explanations of its degradation remain consistent with existing observations. Scientists want to identify an observation that could distinguish them. Operators need to understand approved limits and when to seek specialist help.

A parallel learning workflow supports scientific reconstruction while operation continues within accepted limits. Shared evidence connects scientific and operational accounts. Further inquiry could be justified by understanding alone, without changing the original acceptance decision. New evidence challenging an operating limit, however, must enter the ordinary operational response.

The second scenario concerns public investment in local energy portfolios. Participants explore priorities such as energy output and land preservation using unchanged forecasting assumptions. They may revise their preferences, retain them or reject the offered alternatives. Exploration does not itself authorise a new investment decision, and agreement with the eventual choice is not the test of understanding.

A proposal awaiting empirical tests

The scenarios expose responsibilities that shared storage or an explanation interface can leave unspecified. They do not show that the framework improves learning, decisions or operational performance. The literature synthesis is not systematic, the pattern catalogue is not established as exhaustive, and the authors acknowledge possible confirmation bias in evaluating their own designs.

Independent review and studies with intended recipients remain necessary. The central engineering proposal is nevertheless concrete: distinguish an accepted output, evidence of human understanding and authority to change a decision. None automatically supplies the others.