When Less Data Helps: A Task-Focused Approach to Machine Learning

Keeping every observation seems like a sensible learning strategy. But what if some observations make a limited model less useful for the particular task it must perform? That is the question behind “Forgetting to Improve: Principled Data Removal in Active Learning,” a November 2, 2026 preprint by Manuel Wendl, Erik Englesson, Andreas Krause and Carl Henrik Ek, identified as arXiv:2611.04054v1.

The authors introduce Forgetting to Improve, or F2I, for Gaussian process models. Their central claim is task-specific: when a model cannot adequately represent all the available data, selectively removing observations from its training set can improve predictions in a chosen region. This is not a general argument that smaller datasets are better.

Why additional observations can hurt

Gaussian processes provide predictions together with estimates of uncertainty. The paper distinguishes epistemic uncertainty, associated with limited knowledge, from aleatoric uncertainty, associated with observation noise.

The complication is model mismatch: the model’s assumptions may not describe the process producing the observations. In the authors’ setting, a Gaussian process with one shared noise level can absorb patterns it cannot represent into that noise estimate. Observations from one part of the input space can therefore inflate uncertainty somewhere else, including the region important to the task.

Reducing this fitted noise does not remove randomness from the underlying world. Rather, the method changes how a particular model attributes uncertainty after its training data are revised.

How the forgetting works

F2I starts with a target region: the inputs where useful predictions are needed. Because the correct outputs throughout that region are unavailable, the algorithm does not directly select data by prediction accuracy. Instead, it seeks to reduce the sum of the model’s predictive variances there.

For each observation, the method estimates two consequences of removal: losing information that supports predictions, and changing the fitted noise level. Combining these effects helps protect useful observations that a noise-only removal rule might discard.

The estimates use a first-order approximation, avoiding a separate model refit for every possible removal candidate. F2I removes the observation predicted to reduce target-region uncertainty most, refits the noise estimate and repeats until no further reduction is indicated.

Importantly, an observation need not be corrupted to be detrimental under this criterion. It may describe genuine behavior that the chosen model cannot accommodate while remaining useful for the target task.

What the experiments show

The regression comparisons include synthetic functions and datasets such as Dow Jones, Boston Housing, Yacht and CASP. The authors report means and standard deviations across 100 random training-sample draws, assessing prediction errors, predictive likelihood, calibration and sharpness.

On the Dow Jones task, F2I’s mean squared error was 0.064 ± 0.051, compared with 0.089 ± 0.093 for the standard Gaussian process with a shared noise level. These are benchmark results with substantial variability, not a demonstrated improvement in financial forecasting generally.

The detailed table also shows that F2I does not win every comparison. On Boston Housing, for example, other models achieved lower mean squared errors. Calibration improvements were likewise not uniform across datasets and metrics. In the illustrated Dow Jones example, better target-region calibration came with underconfident predictions across the full domain.

For Bayesian optimization—sequentially choosing evaluations to find a function’s optimum—the authors adapt F2I to a changing region of potentially better solutions. Within each forgetting stage, that region can only shrink, preventing uncertainty created by removal elsewhere from reopening excluded areas.

Across eight test-function settings, the main comparisons used 10 random initializations, five initial observations and 50 subsequent evaluations. The authors report consistently low simple regret, meaning a small gap between the best evaluated value and the optimum. However, an alternative acquisition-rule experiment identifies six-dimensional Ackley as an exception to their better-or-comparable performance claim.

In simulated reinforcement learning, their striking example uses a linear Gaussian process to learn Cartpole Balance. Forgetful RL succeeds where the compared GP-based model-based policy optimization baseline does not, retaining experiences relevant to balancing near upright. The appendix adds nonlinear swing-up experiments; these remain simulated benchmarks, not evidence of deployment on physical robots.

Confidence is not automatically correctness

The theoretical guarantee is conditional: it compares uncertainty reduction with the best removal set of the same size under specified curvature, gradient and realized-improvement assumptions. It does not guarantee the best prediction accuracy or control policy.

The appendix documents cases where these conditions fail, including discrete removals that increase uncertainty despite the local approximation. Repeated model updates also add computational overhead, and the current formulation is specific to Gaussian processes.

The practical lesson is therefore narrower than “forget to learn.” The preprint presents evidence that revisiting which observations a model uses can help under mismatch—but the value of removal depends on the model, the target region and the reliability of its uncertainty estimates.