Better Context Helps a Smaller AI Answer Questions About Scientific Figures

Finding the explanation behind the figure

Answering questions about scientific figures requires more than recognizing bars, lines or labels. Relevant information can be scattered throughout a paper, and some questions demand precise numerical readings. In the RE-SciVQA preprint, researchers propose improving the supporting context supplied to an AI model rather than relying primarily on a larger model.

Their system combines a four-billion-parameter Qwen3-VL model with retrieval-augmented generation, or RAG: a method that supplies selected source passages alongside a question. The authors also adapt the model using QLoRA, a training technique that updates small adapter components while keeping the original model weights frozen in compressed form.

The reported results show both the promise and the hazard of this approach. Carefully selected context helped the adapted model, but adding retrieved passages to the unadapted model made its performance worse.

Selecting context without another ranking model

The pipeline first uses the Docling toolkit to extract structured text blocks from papers, retaining information such as section headings and captions. A text-embedding model converts those blocks into numerical representations that can be searched for similarity to a question or caption.

RE-SciVQA then adds clues to the search. It appends the figure number and relevant section names to the question. For example, questions containing words such as “show” or “display” can receive an “analysis” section cue.

Retrieved passages are subsequently reranked using simple bonuses: 0.15 for a matching figure reference, 0.10 for caption alignment and 0.08 for section priority. The authors say these weights were chosen heuristically; experimenting with alternative weights through grid search produced negligible improvement. The method selects the top two relevant blocks without requiring an additional ranking model.

For adaptation, the researchers freeze the vision encoder and place trainable adapters in the language-model component. They report that training converged within one epoch. The model is instructed to provide direct answers rather than extended reasoning explanations.

What the benchmark shows

The evaluation uses SciVQA, which contains scientific figures from arXiv and ACL papers. Its splits comprise 15,100 training samples, 1,680 validation samples and 4,200 test samples. Questions span seven categories, including visual, nonvisual and unanswerable questions.

The complete system scored 0.739 on ROUGE-1 F1 and 0.738 on ROUGE-L F1. These metrics measure overlap between generated and reference answers; a score of 0.739 should not simply be read as 73.9% factual accuracy.

In the paper’s comparison table, RE-SciVQA substantially outperformed the chart-focused models UniChart, ChartGemma and Matcha-ChartQA. Its ROUGE-1 result was close to the 0.740 reported for an eight-billion-parameter InternVL3 configuration using chain-of-thought prompting. However, a 14-billion-parameter InternVL3 system with adaptation and retrieval remained ahead at 0.798.

The comparison with the underlying Qwen model clarifies where the improvement came from. Without task-specific adaptation, the caption-based baseline scored 0.660. Adding RAG reduced that to 0.583. After adaptation, the model scored 0.732 without retrieval and 0.739 with it. Most of the overall gain therefore appeared after adaptation, with retrieval adding a smaller further improvement.

Extra text can encourage unsupported answers

The sharpest warning came from unanswerable questions. Adding RAG to the unadapted model reduced its ROUGE-1 score in this category from 0.661 to 0.093. The authors attribute this to distracting context that encouraged answers when the appropriate response was to recognize insufficient information.

Adaptation largely reversed that problem. Even then, the adapted model performed slightly better on unanswerable questions without retrieval: 0.915 versus 0.908.

A separate retrieval check measured whether reference-answer strings appeared in the retrieved passages. The enhanced method increased total answer presence from 32.2% to 36.8%, a 4.6-percentage-point increase, or roughly 14.3% relative improvement. This measures answer presence in context, not an equivalent increase in answer accuracy.

Numerical extraction still needs care

The researchers also tested alternative ways to represent chart information. For numerical visual questions, using DePlot to translate plots into tables produced a ROUGE-1 score of 0.493, compared with 0.416 using ChartGemma-generated captions—an approximately 18.5% relative improvement.

That finding does not mean plot digitization improved on the full system: its corresponding score was higher, at 0.541. Rather, extracting numerical data worked better than the tested synthetic-caption approach.

The study supports a focused conclusion: relevant document context and lightweight adaptation can help a smaller model on this benchmark, but extra context is not automatically beneficial. The authors leave broader open-ended scientific visual question answering to future work.