RAGLog looks for hidden faults behind AI-generated answers

A correct answer can hide a broken process

Retrieval-augmented generation, or RAG, combines a language model with information retrieved from external documents. A typical pipeline may rewrite a question, convert it into a numerical representation, retrieve relevant passages, reorder those passages and generate an answer.

A failure anywhere in that sequence can affect later stages. But the final answer does not always reveal the problem: a model might answer correctly from its training knowledge even when retrieval missed the supporting document.

In a November 2026 preprint, Hira Naveed and colleagues introduce RAGLog, a system designed to identify faulty stages from execution logs. Their central argument is that checking answers alone misses useful evidence about pipeline reliability.

In the authors’ 1,500 naturally occurring test runs, 551 contained at least one stage labeled faulty. Of those, 202 still produced a correct final answer. These figures describe the tested systems and questions, not an estimate for RAG applications generally.

Following the evidence through the pipeline

RAGLog first turns heterogeneous API logs into a shared representation of five possible stages: rewriting, embedding, retrieval, reranking and generation. Rules recognize familiar API formats; a language model handles calls or field mappings that the rules cannot resolve.

It then combines three kinds of checks. Rules look for changes such as missing names or dates. Embedding comparisons measure semantic similarity. Targeted language-model checks examine issues such as whether an answer introduces unsupported claims or whether a reranker discarded necessary evidence.

One check explicitly follows the information needed to answer the question. If that information is missing from all retrieved passages, retrieval may be faulty. If it appears only in passages excluded before generation, reranking may be responsible. If the generator receives the information but omits or misstates it, the problem may lie in generation.

Learned classifiers combine these measurements into stage-level verdicts and probabilities. Importantly, “reference-free” describes analysis time: RAGLog does not need a reference answer for each new question, but its classifiers were trained offline using labeled runs.

The tool can ingest existing traces or collect traffic through a logging proxy. The proxy approach avoids modifying application source code, although developers must configure traffic routing and certificate trust.

Better recall, with more false alarms

The researchers assembled RAGLog Faults, a dataset of 3,134 development and test runs from five open-source pipelines across three English-language question-answering datasets. Their main evaluations used 1,500 natural test runs and a selected set of 480 runs subjected to live fault injection.

Using Claude Haiku 5.5 for its language-model checks, RAGLog achieved a pooled stage-level F1 score of 0.73 on natural runs. Claude Code, also using Haiku 5.5, scored 0.58; Codex, using GPT-5.5, scored 0.52. F1 balances precision and recall; these figures are not percentages of answers judged correct.

The advantage came primarily from detecting more faulty stages. RAGLog’s pooled recall was 0.78, versus 0.46 and 0.40 for the agents. Its pooled stage-level false-positive rate was also higher: 4.6%, compared with 1.4% and 1.6%. At the whole-run level, it flagged 13.6% of fault-free natural runs.

On injected runs, the systems were closer: pooled F1 was 0.84 for RAGLog, 0.80 for Claude Code and 0.82 for Codex. RAGLog’s advantage over Codex was not statistically significant. The experiments also showed that an injected disturbance does not necessarily become an observable fault, because later processing can absorb its effects.

The Haiku-based version cost about US$0.005 per analyzed run and had median analysis times of roughly 4.6–4.7 seconds. Those measurements reflect the authors’ setup and provider pricing; agent costs also depended on how questions were grouped into sessions.

What the evidence does not establish

The benchmark’s stage labels came from language-model judges with reference answers and evidence, with another model adjudicating disagreements. The authors did not independently validate test labels through human annotation, so reported effectiveness depends on potentially imperfect automated judgments.

Transfer was also incomplete: excluding each pipeline from training reduced the default version’s pooled F1 on natural runs from 0.73 to 0.67. Natural embedding-stage faults were particularly difficult; that version detected none of the ten labeled cases.

RAGLog currently covers observable stages after ingestion, not ingestion itself or agentic pipelines that dynamically repeat and select stages. Its production effectiveness remains untested. The preprint therefore supports a narrower conclusion: structured, stage-specific log analysis can expose faults that answer-only evaluation overlooks, but its verdicts still require interpretation rather than automatic acceptance.