TRANSMISSION // 2609.33386v2

How Sivia learns to make paper overview figures

Source published

Sivia uses paper-based relation planning and human feedback to generate overview figures. Tests report improved accuracy and presentation, with important limits on scope and evaluation.

AI-generated conceptual illustration for How Sivia learns to make paper overview figures
AI-generated conceptual illustration; not documentary evidence.

A paper overview figure should show how a method works without losing important steps or becoming hard to follow. Sivia is a system designed to make these figures and refine its instructions with human feedback. Its results are encouraging in the settings tested, but do not establish that generated figures are dependable without review.

A paper overview figure can look polished and still misrepresent a method: a component may point to the wrong input, or a step may be missing. A correct figure can also be difficult to read. The Sivia study asks whether a system that plans a figure from a paper and learns from human feedback can improve both scientific fidelity and presentation.

Sivia reads the target paper to identify components, operations, stages and their relationships. It uses overview figures from recent top conferences to inform visual organization, then prepares a detailed prompt for an image generator. In the intended design, the target paper supplies the scientific content; reference figures inform presentation.

The system learns at two levels. “Learning to draw” means revising its instructions, called a skill, using human feedback about accuracy and presentation. “Learning to learn” means revising the updater that proposes those skill changes. Candidate updates are compared using the same parent skill and feedback; people select a candidate and decide whether to admit it. A user profile can also adapt presentation to individual preferences while keeping the scientific content fixed. The underlying language-model and image-generator weights are not updated through this learning process.

Diagnosing figure errors

The researchers first studied 1,964 generated overviews for 445 papers. The papers were selected from recent leading-venue research across several topic areas, including agents, retrieval, reasoning, verification and learning methods. Three raters made 17,973 judgments against source-anchored requirements describing relationships a figure should show. Of judgments that were not correct, about 63% were omissions.

Adding a relation plan extracted from the target paper raised required-relation accuracy (RRA)—the share of required relationships judged correct—by 4.38, 14.52 and 19.85 percentage points across three cohorts. Retrieved reference figures, by contrast, lowered RRA in the reported cohorts. In one cohort, retrieved and random references performed nearly alike, differing by just 0.05 points. Planning also involved a presentation trade-off in one cohort: figures became more complete but slightly less readable. Accuracy and readability are distinct outcomes, and gains in one do not guarantee gains in the other.

Testing learning and personalization

The main learning evaluation used 48 test papers separate from those used to develop the instructions. Each figure was assessed against source-grounded relationships; raters also scored readability and visual quality on five-point scales. The full approach reached 93.54% RRA, compared with 78.23% for the initial skill, 84.17% for a fixed updater and 85.21% for the GEPA comparison. Readability and visual-quality scores were 4.70 and 4.78, up from 3.95 and 4.00 for the initial skill. In a same-parent comparison, the learned updater proposed a skill with RRA 9.90 points higher than the initial updater’s proposal.

A separate preference study involved six users and 48 tasks. Learned profiles received a mean liking score of 4.73, compared with 4.02 for fixed profiles; adoption rose from 34 to 45 tasks. These are results about preference and selection in this small study, not evidence that personalization improves understanding generally.

What remains uncertain

The results suggest that source-derived relation planning and structured human review can improve Sivia’s overview figures under the tested conditions. They do not establish that generated figures are reliably correct without checking. The authors note that a convincing figure can still depict an unsupported mechanism and recommend checking it against the paper.

The main evaluation used GPT-6.1 Sol for all text roles and one image generator for all images. The learning results come from a single learning history, so variation across independent learning runs is unknown. Visual ratings came from six raters in the authors’ research group; those researchers also selected and admitted updates. The preference study involved six users, and all evaluations concerned overview figures for machine-learning papers, limiting conclusions about other fields.

The GEPA comparison omitted Pareto-frontier maintenance and merging; the authors say these require per-instance automatic scores unavailable in their human batch-judgment setup. This qualification matters when interpreting comparisons. The released artifact packages the same-parent comparison, not the other evaluation conditions, so it does not by itself reproduce the full set of results.

The source is an arXiv preprint, and the evidence provided here does not establish peer review. The findings are a reported evaluation, not settled validation. Broader testing across independent runs, users and fields would help clarify how well the approach generalizes.

Automatically matched by shared topic keywords. These links do not imply that the sources agree or establish a consensus.