LadderRAG Searches Across Book Chapters—But Better Retrieval Is Only Half the Story

When one passage is not enough

In Black Beauty, Ginger’s cough appears during a stable fire, after hard hunting and later at a London cab stand. Explaining what that recurring cough means requires connecting three chapters. A search system can instead find another character’s cough and produce a plausible answer about the wrong horse.

That example motivates a research preprint by Dipto Sumit, Syeda Fabiayee Jahan and Farig Sadeque of BRAC University. The authors introduce CrossChapterQA, a benchmark for questions spanning distant sections of books, and LadderRAG, a system designed to retrieve the evidence needed to answer them. The findings are preprint results, not established conclusions.

A benchmark that records where answers live

CrossChapterQA contains 1,853 questions across 14 public-domain English books, including Persuasion, Little Women and Sherlock Holmes stories. Each question identifies two to six required sections, supplies evidence from each and labels the relationship connecting them. Categories include recurring devices, setup and payoff, character or narrative arcs, and reasons for a book’s ordering.

The first and last required sections span roughly half a book at the median. Questions do not name their required chapters, leaving the system to locate them.

Language models generated the questions and two independent reference answers for each. A third model graded those answers blind against the required sections. This provides structured evaluation material, but not human-authored ground truth: the references can still contain errors or omit valid interpretations.

Searching a ladder instead of a graph

Retrieval-augmented generation, or RAG, gives a language model material found by a search system before it answers. LadderRAG organizes that material at three levels: passages, summaries and whole-book cards covering structure, characters, terms, chronology and recurring themes. It builds no entity graph.

The final design breaks a question into parts, searches for supporting passages and adds broader context from summaries and cards. Limits on passages per chapter help prevent one location from crowding out evidence elsewhere.

The authors report an average indexing cost of about $0.41 per book. Final-system answers in the human-evaluation run cost $0.0066 each. These are costs from their runs, not matched comparisons with other systems.

Feedback changed the design, not the model

The first version answered 1,119 sampled questions. An AI judge flagged 191, or 17.1%, as contradicting the references or discussing different events or characters. Crucially, that screen did not count incomplete answers or refusals as failures, so the remaining answers should not be read as an 83% accuracy result.

The researchers redesigned the system using AI judgments, retrieval diagnostics and their own written analyses of mistakes. No model weights changed. Their comparison with learning from human or AI feedback describes an architectural development process, not reinforcement-learning training.

The unsuccessful changes are instructive. One version added a verifier that rejected passages failing its constraints. Retrieval coverage increased, but answers worsened because the verifier sometimes rejected correct evidence. Another version improved retrieval while dropping broad book context, damaging interpretive answers. The retained design combined targeted passage search with restored summaries and cards.

What improved—and what was actually measured

The final evaluation concentrated on the 191 questions the first version had failed. Of these, 178 came from 13 books held out from development; the remaining 13 came from Black Beauty, the development book.

On the held-out questions, location recall rose from 0.36 to 0.79. This measures the share of required sections represented among retrieved passages—not answer accuracy. Questions retrieving none of their required sections fell from 58 to six.

Ten annotators contributed 764 ratings: each answer received three individual scores and one joint score from a four-person English-department panel. The overall mean was 6.48 out of 10, within the rubric’s “fair” band. Thirty-seven percent of answers averaged at least seven.

However, raters differed substantially in strictness. Two more lenient reader scores averaged about 7.7, while the panel averaged 4.7. Absolute agreement was low. The study also collected no human scores for the original answers, so it does not provide a human-rated before-and-after comparison.

Limits on the conclusion

The final system was not evaluated across the full original sample: answers that passed the initial screen were retained from the first version. There were no head-to-head comparisons with GraphRAG, other structured retrieval systems or long-context models.

The books are older English classics likely present in model training data, making memorized knowledge a possible contributor. Annotators saw the references and knew which answers came from the system. Two planned retrieval mechanisms also never activated because of a software bug; reported results reflect that implementation.

The supported conclusion is narrower than reliable whole-book understanding: structured evidence labels helped diagnose retrieval failures, and redesign substantially improved chapter coverage on a selected failure set. Correct interpretation remained a separate, unresolved challenge.