Evaluating what a rock-image classifier predicts—and what its explanations reveal

Looking beyond accuracy

Petrography identifies minerals, rock fragments and fossils in thin sections examined under a polarising microscope. Plane-polarised light, or PPL, reveals features including colour, grain shape and cleavage. Cross-polarised light, or XPL, reveals interference colours, extinction behaviour and twinning. These complementary views provide different evidence for identification.

Thomas James Wood, Oktay Karakuş and Can Eyüpoğlu examined how the released LITHOS image classifier uses these views. Their study, posted as an arXiv preprint, is not presented as a peer-reviewed journal publication. It introduces neither a new classifier nor newly generated image experiments. Instead, it reanalyses saved outputs from an earlier evaluation and adds statistical comparisons of explanation performance.

LITHOS is a dual-stream vision transformer: it processes the two optical views through separate image-processing branches and combines information between them. The researchers asked whether its mistakes follow mineral relationships, whether replacing one view affects predictions more than replacing the other, and whether explanation maps reliably rank influential regions.

Related labels account for many mistakes

The classification analysis covered 105,802 expert-annotated grains across 25 categories. Evaluation used two folds with no thin-section overlap: each fold was assessed using a model trained on the other. This avoids placing grains from the same thin section in both training and evaluation sets.

Mean accuracy was 62.3%, reproducing the published baseline. Macro-F1, which combines precision and recall while giving each category equal weight, was about 0.432. The difference matters because the dataset contains many more examples of some categories than others; overall accuracy does not describe performance equally well across categories.

Many errors involved related labels. Among misclassified plagioclase, microcline and sanidine grains, 72.6% received the generic Feldspar label. Among quartz-family grains, 87.1% received a label within that family, but only 66.6% received their exact label.

These broader agreements are not additional correct answers under the original classification scheme. Nor do they show that the model deliberately expresses uncertainty by choosing a broader label. Other errors crossed group boundaries, and the abundant Rock fragment and Monocrystalline categories together received 51.1% of all misclassifications.

Replacing an optical view

For 706 correctly classified grains, the earlier experiments replaced one optical image at a time with a uniform image retaining each colour channel’s average value. This removes spatial structure, including boundaries and texture, while leaving the other view unchanged.

The researchers measured how much the model’s probability for the expert-labelled category fell. XPL replacement caused a greater average reduction than PPL replacement in 20 of 25 categories. The exceptions included all three quartz categories, Foraminifer and Heavy mineral.

This measures sensitivity to a particular intervention—not the universal importance of XPL or PPL. Uniform replacement also disrupts correspondence between the views and creates inputs unlike the training images. It cannot isolate reliance on a specific optical feature, such as twinning. Only category averages were retained, preventing uncertainty estimates; small differences, including those for quartz, were therefore not statistically resolved.

Testing explanation maps

The study compared five attribution procedures: Grad-CAM, two architecture-specific adaptations of Chefer methods, cross-attention and attention rollout. Attribution maps assign relative importance to image locations. Some procedures target a particular class prediction, while the attention-based baselines do not.

Illustrative maps differed visibly for identical inputs. However, highlighting the central grain does not itself establish that a map identifies regions influencing the decision.

A separate test used 30 correctly classified grains from 29 thin sections, covering six categories. Images were divided into superpixels—groups of neighbouring pixels—and regions were progressively replaced in the order each explanation ranked them. The resulting probability changes were compared with deletion in random order. Higher area-over-curve scores indicated earlier or greater probability reductions.

Method-minus-random differences ranged from −0.043 to 0.057. The researchers resampled whole thin sections to estimate uncertainty, accounting for grains sharing a section. Across all 41 comparisons, every simultaneous 95% confidence interval included zero. The test therefore established neither a reliable ranking of methods nor equivalence to random ordering.

What remains uncertain

The perturbation tests involved one model, one evaluation fold, correctly classified grains only and one replacement strategy. The deletion sample was small and covered only six categories. The illustrated maps came from different grains, so their appearance could not be directly checked against deletion scores. The perturbation experiments were not independently reproduced from source images and checkpoints.

The supported conclusion is narrower than a verdict on explainable AI: classification accuracy, sensitivity to optical views and spatial attribution assess different properties. None can substitute for the others.