PEARL: Prediction Error-Adaptive Regularized Learning for Selective Forgetting under Partial Preference Drift

Forgetting confidence, not rewriting the chatbot

PEARL addresses a selection problem: given several candidate responses, choose the one a particular user is likely to rate most highly. The system receives feedback only on the response it selects. It must therefore balance exploiting its current predictions against trying alternatives that could reveal a better choice.

The authors focus on partial preference drift, where some preferences change while others remain useful. They argue that sliding windows, discounting and restarts can discard valid information alongside stale information because they forget broadly rather than selectively.

Their method builds on neural contextual bandits. A small neural network predicts rewards using user and response features, while a matrix tracks confidence across learned feature directions. Standard updates accumulate evidence and reduce uncertainty. The problem, as the authors frame it, is that confidence built from old observations can persist after the preferences behind those observations change.

PEARL adds two responses to prediction errors. First, it tracks each user’s recent errors and increases that user’s exploration bonus when predictions become less reliable. Second, when an observed error exceeds the running average, subject to an additional threshold, it increases uncertainty along the selected response’s learned feature direction. A numerical cap prevents that uncertainty from exceeding its initial level.

This is selective forgetting at the uncertainty level, not direct editing of a named preference such as factuality. The reward network learns from subsequent feedback; the underlying language models remain frozen. The authors describe the selector as roughly 100,000 parameters, operating on precomputed features in milliseconds on a CPU.

What the experiments report

The main evaluation uses PRISM, which contains ratings from 1,500 participants on responses from 21 language models. The experiments retain interactions with four candidates and sufficiently different ratings, leaving 5,011 interactions, or 18.5% of the original interactions. Ratings are normalized to a zero-to-one reward scale.

Crucially, the preference changes are imposed by the researchers. After pretraining, they invert the recorded rewards for selected users, turning previously preferred responses into poorly rated ones. In the moderate setting, half the users undergo this inversion.

After ten evaluation epochs across 20 random seeds, the authors report average reward of 0.641 for PEARL, compared with 0.601 for an online-updated greedy selector and 0.616 for an error-adaptive neural bandit without uncertainty injection. These are gains of 4.0 and 2.5 points on a 100-point reward scale, not percentage improvements. The paper reports statistical significance below 0.001 for these comparisons.

With no preference inversion, PEARL scores 0.681 against greedy selection’s 0.684; the reported difference is not statistically significant. A separate ten-user simulation also favors PEARL under complete preference inversion. In another PRISM protocol, rewards transition toward inverted values over three epochs and then stabilize; PEARL finishes 3.6 reward points ahead of greedy selection.

Where the evidence stops

These are controlled adaptation experiments, not observations of users naturally changing their minds. Even the gradual protocol blends original and inverted ratings rather than collecting new longitudinal feedback. Continuous, indefinitely changing preferences are not evaluated.

The headline results also follow repeated passes through recorded interactions. At the moderate inversion level, PEARL’s first-epoch advantage over greedy selection is not statistically significant. Under mild inversion, greedy selection leads initially. The reported benefit therefore should not be read as an immediate improvement for every user or every interaction.

Filtering favors interactions with clearly separated ratings, although the authors report that the ordering persists under looser thresholds. Their theoretical guarantees require assumptions about reward noise, calibrated uncertainty and bounded preference variation. The claimed one-interaction response describes the error tracker under stated conditions, not complete recovery of the recommendation policy.

The authors also acknowledge that the trigger is reactive, long-horizon behavior remains open, and binary or implicit feedback needs further adaptation. PEARL’s supported contribution is narrower than a general solution to changing tastes: it offers evidence that targeted uncertainty updates can improve recovery in these controlled response-selection settings.