IDSISA reorganizes training data to make AI forgetting more selective

Organize first, forget later

In a November 4 preprint, Deeraj SK, Sadhana Devarajan, Tanmoy Hazra and Anupam Shukla introduce IDSISA, an extension of the SISA machine-unlearning framework. The manuscript is submitted for possible journal publication; it does not establish peer-reviewed findings.

SISA divides training data into independent partitions, called shards, and trains a separate model on each. Training proceeds through slices with saved checkpoints. When records must be removed, affected models restart from checkpoints preceding those records and retrain without them. Predictions are then aggregated across models.

IDSISA changes both the organization of those partitions and the retraining process. Its results suggest improvements over the authors’ SISA baseline, but not a universal shortcut to privacy compliance or equivalence to training anew.

Two additions to sharded training

First, IDSISA estimates how strongly individual examples affect predictions. An early-stopped proxy model supplies features and gradient-based influence estimates. Feature reduction and clustering guide shard assignment, aiming to concentrate examples that might be deleted together while keeping partition sizes balanced.

This organization is most useful when deletion requests align with recognizable structure, such as an image class. It does not establish that influence scores can predict arbitrary future deletion requests.

Second, affected shard models learn from unaffected ones during retraining. These teachers provide probability distributions rather than just class labels, a technique called knowledge distillation. The default configuration equally weights the ordinary classification loss and the distillation loss, using temperature 2.0 to soften teacher predictions.

The intended benefit is to recover predictive behavior lost when a model trains on only a small partition. If no unaffected teachers remain, the method retrains without distillation.

What the experiments actually show

The main experiments report averages over five random seeds, using PyTorch 2.2 on an NVIDIA A100. They cover ResNet-18 on CIFAR-10 and CIFAR-100, BiLSTM and temporal convolutional models on IMDB, and a fully connected network on Fashion-MNIST. Image and sentiment experiments use 10 shards; Fashion-MNIST uses 15. The main performance tables concern single deletion requests.

On CIFAR-10, deleting the 5,000 training images in the cat class reduced accuracy on that class from SISA’s 18.6% to IDSISA’s 12.3%. Retained-class accuracy increased from 83.2% to 84.7%. Reported preservation loss fell from 1.6 to 0.8 percentage points: the basis for the paper’s 50% relative improvement claim, not a 50-point accuracy increase. Unlearning took 81.2 rather than 86.7 minutes, while initial training increased from 59.9 to 66.1 minutes.

For BiLSTM on IMDB, removing 1,000 positive reviews increased the reported forgetting gap from 2.06 to 3.94 percentage points. That is the roughly 91% relative gain highlighted by the authors, not evidence that 91% of training influence disappeared. Retained-class preservation change improved from −3.10 to +0.65 points.

Using the same deletion target, the temporal convolutional model’s forgetting gap increased from 38.48 to 41.96 points. However, both methods’ positive-class accuracy fell below 50%; the authors attribute this to shard-level class-imbalance collapse. Fashion-MNIST deletion removed 6,000 examples across all 10 classes, with unlearning taking 570 seconds versus SISA’s 534 seconds.

CIFAR-100 exposes a clearer limitation. Removing approximately 2,500 large-carnivore examples produced zero forgotten-superclass accuracy under both methods, against overall accuracy near 21%. The authors explicitly say this cannot be credited to successful unlearning. Retained-class accuracy improved modestly, from 21.15% to 21.89%.

“Exact” has a restricted meaning

The paper defines its guarantee as causal exactness: repaired model computations must not receive forgotten examples or contaminated teacher parameters. It does not claim that distilled models are statistically indistinguishable from models trained from scratch on retained data.

A separate output-deviation measure averaged 0.011, 0.008 and 0.013 for the CIFAR-10, BiLSTM and Fashion-MNIST models. Their 99th-percentile values were higher, at 0.024, 0.019 and 0.027. The proposed 0.015 mean threshold is not a worst-case guarantee. Its theoretical justification is explicitly heuristic, relying on convexity surrogates, illustrative constants and an additional shard-independence assumption.

The guarantee also does not cover all preprocessing information. The evaluated influence scores, proxy features and clustering metadata were computed using the original dataset and retained afterward. A proposed differentially private alternative was not implemented.

Repeated deletion can spread the work

Distillation creates dependencies: a teacher may transmit information that a later request requires removing. IDSISA therefore tracks teacher–student relationships and retrains every reachable affected model, not just shards containing newly deleted records.

In an eight-request stress trace, cascades occurred five times and required all 10 shards twice. That trace used full-dataset warm starts and reduced retraining budgets, so it characterizes dependency costs rather than independently validating clean deletion. Transformers were not evaluated, and code release is promised upon acceptance. The practical finding is a task-dependent trade-off between forgetting metrics, retained accuracy, setup costs and potentially expanding repair work.