When visual AI learns the answer but loses the thread

From seeing shapes to following rules

A missing square in a pattern presents two problems: identify what is visible, then work out what belongs in the gap. A vision-language model, which processes images and text, can describe the shapes correctly yet apply the wrong rule. Even selecting the right answer does not establish that its explanation is sound.

In a preprint, researchers at Yildiz Technical University investigate this distinction using visual reasoning questions presented in Turkish. Their results offer a qualified finding: one fine-tuning recipe substantially improved reported answer accuracy, but other recipes led to repetitive explanations and frequent empty outputs. The study does not establish whether models “think” in a human sense; it examines answers and generated explanations on particular tasks.

Building a Turkish puzzle collection

The researchers assembled material from four sources: PuzzleVQA, MM-IQ, Visual Puzzles and preparation questions for Turkey’s BİLSEM Science and Art Centers. Their dataset table lists 5,240 examples, including 400 test questions—100 from each source.

English-source text was translated into Turkish using language models. An image-processing pipeline combined the question, puzzle image and answer choices into a single vertically arranged image. Solving a question therefore required interpreting both visual patterns and text embedded in the picture.

Training also needed worked solutions. Some source datasets already supplied explanations; others did not. The researchers generated synthetic solutions using Gemini 3 Pro and Qwen3-VL-Flash. Crucially, these generators received the known correct answer as guidance. Outputs suggesting an incorrect answer were filtered in parts of the workflow, but an answer that matches the key does not guarantee a valid explanation.

The team fine-tuned Qwen3VL-8B and Gemma3-12B using supervised fine-tuning: training models to imitate supplied examples. They used LoRA, a method that adapts a comparatively small set of parameters rather than updating the entire model.

A gain that depended on the training recipe

The strongest reported fine-tuned configuration was Qwen3VL-8B trained on a mixture of PuzzleVQA and MM-IQ solutions generated by Gemini 3 Pro. Its reported overall accuracy rose from 24.7% without this fine-tuning to 48.8%.

The improvement was uneven. That configuration scored 72% on PuzzleVQA, but 26.7% on BİLSEM and 22% on Visual Puzzles. Those differences caution against treating the overall increase as a broad solution to visual reasoning.

Adding more synthetic material did not reliably help. Combining Gemini-generated and QwenFlash-generated solutions reduced reported overall accuracy to 32.3%. Training on QwenFlash solutions alone produced 30.5%.

The researchers also tested MultiSFT, which added two alternative generated solution paths per problem and filtered invalid answers. On the combined-source recipe, this recovered accuracy to 40.5%—better than ordinary mixed-source training, but below the best Gemini-only recipe. Filtering final answers alone did not deliver a comparable recovery.

When explanations stop progressing

The failures were not simply wrong answer choices. Some models repeated a partial rule without reaching a conclusion; others returned no output.

For the strongest Gemini-only recipe, the paper reports 1% empty outputs and 9% repetition. With ordinary mixed-source training, those figures rose to 42.25% and 52%, respectively. MultiSFT reduced them to 6.25% and 13.25%.

The authors describe these behaviors as model collapse. Here, that term refers to generation failures observed in the tested configurations, not demonstrated permanent damage across all model capabilities. The experiments associate training recipes with instability; they do not isolate a single underlying cause.

Why the explanation needs its own scrutiny

To assess more than answer accuracy, the researchers used Gemini 2.5 Flash as an automated judge. It scored explanations for logical consistency, clarity and agreement with reference solutions, with model identities hidden in paired comparisons.

That adds information about the written rationale, but it is not direct access to a model’s internal computation. A judge can favor reference-like wording or overlook errors, and the paper does not report a human-validated error rate for its judgments.

Other reporting issues warrant caution. The methods describe an initial 80–20 split, while the dataset table lists different final training and test proportions without fully reconciling them. Results tables omit a separate MM-IQ score despite its inclusion in the stated test set, leaving the overall aggregation unclear. Uncertainty estimates and repeated-run variability are not reported.

The useful lesson is narrower than proof of genuine logical deduction: on these Turkish visual tasks, the choice of worked examples mattered greatly. More training material was not automatically better, and accuracy alone concealed important failures in whether models produced usable explanations.