Certified Circuits reports leaner AI explanations, but its stability claim has a gap

Why example selection matters

Mechanistic interpretability investigates how neural networks produce outputs. Researchers often seek a circuit: a small subnetwork that preserves a target behavior when other components are disabled. Discovery starts with a concept dataset, such as photographs of an object category or prompts testing a language task.

Different examples can yield different circuits. Certified Circuits addresses this sensitivity by wrapping an existing discovery algorithm with randomized data deletion and component-level voting. The authors claim that their resulting circuits remain unchanged under bounded dataset edits. The supplied proof, however, establishes a narrower form of stability.

How the framework selects components

The method repeatedly deletes examples independently, runs circuit discovery on each remaining dataset and counts how often each neuron or edge is selected. A component is marked in when its inclusion probability exceeds a threshold, out when its exclusion probability exceeds that threshold, and abstain otherwise. Only components marked in enter the final circuit.

The experiments use 1,000 deletion samples for certification, plus independent preliminary samples to select which decision to test. Multiple-testing correction controls statistical errors across components. The authors report 99.9% confidence, conditional on their framework’s assumptions; that statistical confidence does not repair the distinction between the output being proved stable and the output actually used.

The reported edit radius counts example insertions, deletions or substitutions. Default vision settings produce radius one; selected language settings produce radii eight to 59. These are the authors’ reported certificate radii, not established bounds on unchanged final circuit membership.

Where the guarantee falls short

Appendix A proves invariance of a component’s non-abstaining binary majority label: whether inclusion or exclusion has greater probability. The circuit-construction rule instead requires the winning probability to exceed a higher threshold. Keeping the same majority does not ensure that the component still clears that threshold after an edit.

A counterexample illustrates the difference. Suppose deletion probability is 0.6 and the threshold is 0.95. Let a permissible base algorithm exclude a component only when one distinguished example is absent and three other examples are present. Its inclusion probability is initially 0.9616. Deleting the distinguished example from the dataset lowers that probability to 0.936.

The component therefore changes from in to abstain after one edit, despite the reported radius being one. Its binary majority remains inclusion throughout. Because abstained components are omitted, final circuit membership changes. Thus the proof does not establish the stronger invariance asserted in Theorem 3.1 and the paper’s circuit-level descriptions. This gap does not itself invalidate the reported performance measurements.

What the experiments found

Vision experiments used ResNet-50, ResNet-101 and ViT-B/16, with main discovery datasets containing 50 images per class across 100 randomly selected ImageNet classes. Language experiments used GPT-2 Small and EAP-IG edge attribution on indirect object identification, a harder distractor-rich variant, and an adapted Greater-Than task.

For ResNet-101 on ImageNet, reported peak circuit-class accuracy rose from 83% to 95%, while circuit size fell 52%. On ImageNet-A, accuracy increased from 60% to 94%, with 15% fewer components. The paper’s 56% improvement uses unrounded values and is relative, not a percentage-point increase.

Both methods reached 100% accuracy on all three in-distribution language tasks, but the proposed method retained fewer edges. Shifted Greater-Than prompts showed an 80% edge reduction at matched accuracy. Shifted IOI-Hard accuracy rose from 85% to 91%, with little size reduction. ViT-B/16 gains were smaller: peak accuracy differed by at most approximately two percentage points.

Evaluation limits

Vision in-distribution evaluation reused discovery images, so those results do not demonstrate independent test-set performance. Language prompts were held out, but certification settings were selected separately for each task and evaluation split using peak performance.

Vision accuracy concerns class-specific circuits, not a demonstrated classifier that selects among all classes without a supplied target circuit. Dataset-edit stability also differs from accuracy on unfamiliar inputs or a complete explanation of model reasoning. Larger-model and multimodal validation remains open.

Publication and source provenance

The supplied paper carries a proceedings notice for the 43rd International Conference on Machine Learning, PMLR 306, 2026; the original v4 arXiv landing record also identifies ICML publication. A separate check of the ICML proceedings listing has not been performed here.

Verification against the two supplied original arXiv landing HTML records confirms the metadata’s paper title, all five authors in order, v4 abstract, version-specific URL https://arxiv.org/abs/2602.22968v4, and original v1 timestamp 2026-02-26T13:07:31.000Z. The submission histories date v4 to November 4, 2026. The original v1 record contains a different abstract, including earlier performance claims; it must not be conflated with v4. These metadata checks are complete against the supplied original records and are distinct from the unperformed proceedings-listing check.