Can comparison data help AI reward models respect Pareto optimality?

When a score contradicts its principles

Many AI alignment methods learn from comparisons between possible responses. A response might be judged more helpful or less toxic, and a reward model is trained to score it higher. That score can then guide a system toward responses that receive better evaluations.

But a single score must combine judgments made according to multiple principles. What if one response is preferred by every principle, yet the reward model gives it a lower score? The paper studies this possibility using Pareto optimality (PO). In this setting, PO means that if response A is preferred to response B by all the principles under consideration, A should not receive a lower reward. For example, if A is both more helpful and less toxic than B, a PO-respecting reward model should not rank B above A.

The authors ask whether this contradiction can be avoided by making reward models more flexible, or by changing the comparisons used for training.

A mathematical model of principles

The analysis treats each principle as a “voter” that compares responses. Helpfulness and toxicity, for instance, may each express a preference between the same pair. The paper represents these preferences as directions in a fixed embedding space—a mathematical space in which responses are represented as points. This simplifying assumption is motivated by the Linear Representation Hypothesis, the idea that some concepts can be represented by directions in a model’s internal space.

The authors analyze loss-based training on pairwise comparisons, a broad setup used in preference-based reward learning. They consider linear reward models and more expressive alternatives: polynomials of any fixed degree and one-hidden-layer ReLU neural networks of fixed width. These are mathematical analyses, not experiments with a deployed language model.

Greater complexity is no guarantee

The central finding is that added model complexity does not, by itself, ensure Pareto optimality. The authors show that PO can still fail for polynomials of any fixed degree and for one-hidden-layer ReLU networks of any fixed width of at least two, given suitable finite parameter bounds. In plain terms, a richer mathematical model can still give a higher score to a response that every represented principle considers worse.

The paper also proves a route to recovering PO for linear reward models. This result holds under specific assumptions in an idealized setting: comparison directions are uniform, and comparisons have equal lengths. The authors suggest dataset design as a possible way to address the problem while retaining loss-based training. The abstract does not establish how these conditions would be achieved with real-world comparisons.

What the result does—and does not—show

These are theoretical results under stated assumptions, not evidence that a particular training dataset or real-world assistant violates PO. Nor does the abstract establish that ordinary comparison datasets meet the conditions needed for the linear-model result. Whether and how such comparison data could be constructed or used in practice remains an open question based on the available evidence.

The result is also narrower than a general claim about AI alignment. Pareto optimality captures one consistency requirement among principles; satisfying it would not determine which principles should be chosen, how they should be interpreted, or how other alignment goals should be balanced. The representation of principles as fixed directions in an embedding space is another important assumption: the abstract does not show how well it describes real preferences.

The available evidence is the paper’s abstract, not its full text, so proof details and further limitations cannot be independently assessed here. The paper is an arXiv preprint, and the supplied record does not establish that it has undergone peer review. Its contribution is best read as a mathematical result and a research direction: within this model of preference learning, expressive reward functions alone do not enforce Pareto optimality, while structured comparison data can recover it for linear rewards under idealized conditions.