A New Test Asks Which Data Maps Preserve Neighbors—and the Bigger Picture

Beyond familiar labels

Dimension-reduction methods turn complex records into simpler representations, often two-dimensional maps. Researchers use them to inspect tissue images, single cells, customer purchases and text collections. A familiar rule of thumb describes t-SNE and UMAP as preserving local neighborhoods, while principal component analysis, or PCA, preserves the bigger picture.

In the preprint Quantifying Local and Global Fidelity of Nonlinear Embeddings, Henry Han argues that these labels should be tested on the actual data. His proposed framework measures local and global fidelity using the same underlying statistic, then compares two fitted maps with a statistical test.

The central finding is not that one method always wins. Rather, the reported results show that the answer depends on the dataset, the reference geometry and the scale being examined.

One curve, two readings

The framework starts with an existing neighborhood-preservation curve. For each point, it asks how many of its nearest neighbors in the reference representation remain among its nearest neighbors in the map. Repeating that calculation for progressively larger neighborhoods produces a profile.

Chance correction matters because large neighborhoods overlap even in an uninformative map. After correction, a random map scores zero in expectation, while perfect neighborhood preservation scores one.

Han defines local fidelity, L, as the corrected overlap for each point’s 10 nearest neighbors. Global fidelity, G, averages the same statistic over neighborhoods containing 30% to 50% of the dataset.

The paper supplies a mathematical interpretation for the profile’s midpoint: it measures agreement about which half of the other points is nearer and which half is farther away. Under stated sampling assumptions, this connects to a rank correlation of distances. The contribution therefore builds on an established curve rather than inventing neighborhood overlap itself.

Testing whether maps exchange strengths

The proposed LG reversal test asks whether one map has higher local fidelity while another has higher global fidelity. Both directional comparisons must pass before the test declares that division of strengths.

To account for dependence between nearby points, the procedure groups points into blocks and repeatedly leaves out one block when estimating uncertainty. It also estimates a crossover: the neighborhood scale where the two profiles first exchange order, with a confidence interval.

Han proposes using this crossover to guide which map an analyst consults for small-neighborhood questions versus broader organization. It is a recommendation derived from fidelity profiles, not a demonstrated guarantee that every downstream task will improve at that threshold.

Where the usual rule holds—and fails

The main evaluation covers 156 maps: seven methods on each of twelve large datasets, plus 72 maps from parameter experiments on MNIST digits. The largest dataset contains about 1.38 million single nuclei.

Across those maps, G has reported rank correlations of 0.89 with distance rank correlation and 0.96 with random-triplet accuracy, two established global-quality measures.

The reversal test supports the familiar t-SNE-local, PCA-global pattern on seven of the twelve large datasets. PCA has the highest G on six, while PaCMAP leads on four.

But t-SNE outperforms PCA at both measured ends on Instacart customer purchases. A smaller Online Retail dataset shows a similar pattern. On synthetic Swiss-roll and S-curve datasets, evaluated against their unrolled geometry, neighborhood-based methods also preserve global relationships better than PCA.

These results underline a qualification: global fidelity is always relative to the chosen reference distances.

A million-passage example

For one million MS MARCO passages represented by BGE-base vectors, the test compares PCA with SR-tSNE2, a rank-refined t-SNE method. It reports SR-tSNE2 as more local and PCA as more global, with a reversal p-value of 0.000026.

The estimated crossover is 150,200 passages, with a 95% interval from 78,900 to 221,500. Han proposes the neighborhood-focused map for inspecting passages around a query and PCA for inspecting topic organization.

Importantly, retrieval in this example still operates in the original 768-dimensional representation. The two-dimensional maps are inspection views, not replacements for that retrieval system.

Important qualifications

The sampling results are encouraging but specific: for PCA and t-SNE, mean G estimates from 2,000-point samples differed by less than 0.01 from estimates using 8,000 points on every large dataset tested. Fixed-neighbor L cannot be estimated the same way without changing its effective scale.

Statistical calibration is also uneven. The global test’s average false-positive rate was close to its nominal 5%, but the local test reached 23.6% on one dataset under the paper’s simulated null. The very small reported local p-values do not establish universal calibration.

Finally, the test compares already-fitted maps. New random starts can alter global layout, and changing the number of blocks changed some declarations. The preprint offers a quantitative check on mapping assumptions—not a universal certificate of faithfulness.