AI reliability starts with asking what counts as success

What should count as success when an AI gives the right answer but cannot identify why it is right—or follows a rule that quietly leaves cases unresolved? This week’s selected papers approach reliability at different levels: supplied facts, completed work, internal model behavior and the rules used to judge outputs. Their shared topic is not evidence of agreement: they test different things.

Knowing facts is not the same as using evidence

Rel-Bloom isolates one step behind retrieval-augmented generation: answering from supplied information. Instead of testing document search, it puts facts directly into prompts, using 450 multiple-choice questions built from 30 relations, across six languages. Its tasks distinguish applying information, checking relations and identifying which property justifies an answer. Gemini 3.1 Pro averaged 99.5% across the other four task levels but 60.7% on that last, justification-focused level. This is answer selection, not a demonstration that a model can explain its reasoning faithfully. The authors also caution that item construction may contribute to differences between levels. Rel-Bloom

FrontierPhysics asks a broader question: can an agent carry a research project through literature review, investigation and a manuscript? Its 56 physicist-designed tasks combine numerical checks with reviewer agents applying expert-written criteria. Across 26 configurations, the leader, Opus 5.5 in Claude Code, passed 46 of 168 attempts. A separate manual audit covered 168 attempts by GPT-6 Astra xhigh, identifying weak rigor and missing expert intuition as leading problems for that configuration. Correcting evaluation errors raised its passes from 25 to 34. Other models or model–tool combinations may fail differently; this audit does not establish the leader’s main failure modes. The distinction between having information and using it well echoes Rel-Bloom, but open-ended research and multiple-choice questions cannot establish a shared cause. FrontierPhysics

Useful to whom—and judged by which rules?

LifeVal shifts the target from scientific work to personalized consumer tasks, such as moving or job hunting. Tasks draw on creators’ real ChatGPT histories and personal context; survey-reported willingness to pay helps prioritize activities. Like FrontierPhysics, it uses people with relevant experience to define success. Unlike physics outputs that permit numerical checks, personalized usefulness requires judgments about a person’s circumstances. Its survey was US-centered and English-language, and willingness to pay is a stated preference, not observed long-term benefit. Its construction therefore offers a way to ask about usefulness, not proof that assistants already deliver it. LifeVal

But explicit criteria need scrutiny too. One Scale, Four Texts translates four diabetic-eye-disease grading documents into formal rules and checks their consequences. It finds gaps and disagreements that ordinary readers may silently repair. In its language-model experiment, deliberation reduced silent repair within two models without eliminating it. This adds a different reliability problem: an apparently sensible answer may hide ambiguity in the instructions themselves, rather than fail to follow clear evidence. One Scale, Four Texts

That study’s photograph-based comparisons also depend on assumptions about retinal thickening, which color photographs cannot show. Leaving that finding unspecified makes the reported referral disagreements involving England undecidable. Neither those conditional disagreements nor the small language-model experiment establishes how deployed clinical systems behave. One Scale, Four Texts

Predicting behavior is different from making it robust

Two papers look inside models. Context–Answer Metamodels trains predictors of averaged internal activation patterns—the numerical activity produced during an answer—from activity recorded while reading the prompt. Forecasts retain signals useful for predicting behavior, but accurate averages do not guarantee accurate variation between answers: the Gemma 4 experiments report poor variance capture. Behavior labels also lack extensive human validation. Context–Answer Metamodels

Addition by Subtraction instead intervenes during training: it progressively disables identified safety-critical neurons—internal computational units—and trains the remaining model to maintain safer behavior. Across four models, the authors report stronger resistance to several attack types while largely preserving general capabilities. They do not directly measure the absolute spread of safety behavior through the network, and neuron identification is imperfect. Forecasting behavior and hardening refusal mechanisms are complementary research questions, not evidence that either method solves the other’s limitations. Addition by Subtraction

What remains unresolved

Across this selection, the practical question is which success measure survives a change in task, user, wording or attack. These studies do not supply a common yardstick. The evidence packet contains abstracts and selected extracted text, not complete papers; extraction can disrupt tables and reading order. Abstract-only evidence cannot establish unreported methods or checks, and missing excerpted details are not necessarily missing from a paper. These are arXiv v1 sources: preprint findings warrant caution rather than an assumption of peer review. Rel-Bloom’s supplied text explicitly identifies it as a COLM 2026 conference paper. Rel-Bloom