A balanced AI score is not immune to unbalanced data

A model that makes yes-or-no predictions can fail in two directions: it can miss a genuine positive or raise a false alarm. Evaluating it therefore means deciding how to combine correct detections, correct rejections and both kinds of mistake. The Matthews correlation coefficient, or MCC, incorporates all four outcomes. But that does not make it immune to changes in how common each class is.

That distinction is central to a retrospective by Jesús S. Aguilar-Ruiz, of Pablo de Olavide University in Spain. The arXiv preprint combines a historical survey with mathematical analysis and graphical comparisons. Its message is not that MCC should be abandoned, but that its reputation as a balanced score needs a precise explanation.

From protein structure to prediction scores

The survey traces MCC’s name to Brian Matthews’s 1975 work comparing predicted and observed secondary structures in a protein. For each amino-acid residue, a prediction could identify a structure such as a helix—or say it was absent. Comparing those predictions with observations produced the same four categories used in modern binary classification.

These are true positives, false negatives, false positives and true negatives. Together they form a confusion matrix: a compact account of what the system got right and wrong.

MCC combines those counts into a value between minus one and plus one when its denominator is nonzero. Plus one represents perfect agreement; minus one represents complete reversal of the labels. The survey explains that MCC is exactly Pearson’s correlation calculated between the actual and predicted binary labels. Its mathematical ancestry therefore predates its association with Matthews.

The coefficient treats the two class labels symmetrically: swapping positive and negative labels in both truth and predictions leaves the score unchanged. Unlike F1, its formula also explicitly includes correct negative predictions. Those properties help explain its appeal, but neither establishes that it is independent of class proportions.

Same detection rates, different score

The paper makes that limitation concrete with a small numerical example. Start with three actual positives and four actual negatives. The classifier correctly identifies two positives and three negatives, missing one positive and producing one false alarm. Its MCC is approximately 0.417.

Now multiply the negative-class counts by five while leaving the positive-class counts unchanged. There are still two correct positive detections and one miss, but now fifteen correct negative decisions and five false alarms. The positive detection rate remains two-thirds; the negative detection rate remains three-quarters. Nevertheless, MCC falls to approximately 0.305.

Nothing in this construction changes either class-specific success rate. What changes is the mixture of cases being evaluated. MCC’s denominator includes totals for both actual and predicted classes, so changing that mixture can change the correlation.

This demonstrates that MCC is not generally prevalence-invariant: holding the conditional performance rates fixed does not guarantee a fixed score. A change in MCC between differently composed datasets therefore need not, by itself, demonstrate a change in those rates.

Seeing what one number conceals

To explore this behavior more broadly, Aguilar-Ruiz uses contour plots. The horizontal axis gives the false-positive rate—the fraction of actual negatives incorrectly flagged. The vertical axis gives the true-positive rate—the fraction of actual positives detected. Each contour joins combinations that receive the same evaluation score.

The paper compares accuracy, MCC and the geometric mean of sensitivity and specificity, known as G-mean, under several class distributions. Accuracy’s contours change with class proportions, reflecting its dependence on the dominant class. MCC’s contours also deform. G-mean’s remain unchanged because it is calculated directly from the two class-specific rates rather than their frequencies.

That is a particular mathematical advantage, not proof that G-mean is universally the best evaluation measure. The application still determines which errors matter and what information an evaluation must preserve.

What the analysis does—and does not—establish

The survey also cautions against reading MCC equal to zero as evidence that a model literally guesses randomly. With a nonzero denominator, zero MCC corresponds to equal true-positive and false-positive rates. A deterministic classifier can satisfy that condition too.

There are boundary cases as well. If every prediction has the same label, MCC’s denominator is zero. Some software reports zero by convention, but that is not the same as a mathematically defined zero correlation.

This is a preprint, not a peer-reviewed validation of a new prediction system. Its central evidence consists of derivations, constructed examples and visualizations rather than a deployment trial. Its analysis focuses on binary classification and confusion-matrix-based scores; it does not settle how best to assess probability estimates or multiclass systems.

The practical lesson is to keep the underlying counts and class-specific rates visible. MCC can be useful without being a universal verdict—and a score that treats class labels symmetrically can still respond to how many examples of each class enter the test.