Choosing Which AI Preferences to Label—and When Disagreement Helps

A comparison is not a single question

Two answers to the same prompt can rank differently depending on what matters. One might provide more information while another better protects privacy. Labeling a response pair for helpfulness therefore does not necessarily reveal which answer is preferable for harmlessness.

In Objective-Conditioned Active Preference Learning for Multi-Objective Alignment, Vu Duc Anh, Thong Nguyen, Hoang Pham, Ponhvoan Srey and Luu Anh Tuan propose OGC-DPO, short for Objective-conditioned Gradient Coverage DPO. Their arXiv preprint treats each combination of a response pair and an objective as a separate labeling decision.

The distinction changes how a fixed annotation budget is spent. Instead of selecting a pair and requesting judgments on every objective, the method can ask only about the objectives expected to provide useful training information. The same pair can still be selected for several objectives when warranted.

Selecting expected changes, not just uncertain labels

OGC-DPO builds on Direct Preference Optimization, or DPO, which trains a language model using preferred and dispreferred responses. For each objective, the framework maintains a small trainable adapter on a shared frozen model backbone. These adapters estimate which response an annotation oracle would prefer.

Selection combines two signals. The first measures disagreement between the policy’s current implicit preference and an objective-specific estimate. The second measures whether the associated projected policy-gradient direction is already well represented among queries acquired for that objective.

In accessible terms, the method looks for judgments expected to change the model in directions that have not already received much supervision. It tracks coverage separately for each objective: a comparison can be redundant for helpfulness but still informative for honesty.

Queries are selected sequentially, with coverage updated after each choice to discourage repetitive selections. The first round is random, supplying labels both for policy training and for initializing the objective-specific estimators.

This is not simply uncertainty sampling. If the policy and an estimator both assign equal probability to either response, their disagreement is zero, even though the label is maximally uncertain. The authors instead target the estimated mean training update.

What the experiments found

The researchers evaluated three model families: Llama-3.2-3B-Instruct, Gemma-3-4B and Qwen3.5-9B. They used PKU-SafeRLHF with two objectives, UltraFeedback with four and HelpSteer with five.

Each method received 300 objective-specific labels per round over six rounds, totaling 1,800 labels. OGC-DPO’s random initialization used that same budget, not additional seed labels. Comparisons included random selection, ActiveDPO and Active Preference Optimization, with a shared policy-training procedure.

The reported results give OGC-DPO the highest average reward in all nine model–dataset combinations. On PKU-SafeRLHF, Qwen3.5-9B reached an average reward of 10.72, compared with 9.37 for the strongest baseline on that metric. For Llama-3.2-3B on the same dataset, the lowest objective reward reached 7.66, versus 5.30 for the strongest baseline.

A complementary GPT-4o assessment compared trained responses with initial-policy responses. In that Llama setting, OGC-DPO’s win rate was 76.4%, compared with 66.5% for the strongest baseline. However, OGC-DPO did not lead every weakest-objective or win-rate comparison across the full evaluation.

Why keep conflicting preferences?

The authors also tested a variant that excluded pairs whose reward-model preferences conflicted across objectives, replacing them with non-conflicting queries under the same label budget. Average reward fell in all nine settings.

That supports a narrower conclusion than saying all disagreement helps: blanket filtering can remove useful supervision. OGC-DPO does not explicitly reward conflict. Rather, when objective estimates disagree, the policy cannot match both, potentially making at least one objective-specific query informative. Coverage still determines whether that signal is novel.

What the evidence does not establish

Although acquisition does not directly consult an external reward model, the experiments use external reward models to supply labels and evaluate held-out responses. The results therefore demonstrate gains against automated preference proxies, not confirmed improvements in human judgments. GPT-4o judging adds another automated perspective, not a human evaluation.

The study covers English datasets, models with three to nine billion parameters, two to five objectives and short acquisition runs. Selection also depends on estimator accuracy. Equal label budgets do not establish equal computation costs, and the accounting assumes each objective-specific judgment costs the same.

Finally, the paper’s mathematical coverage guarantee is not a guarantee of better generalization. The practical contribution is an experimentally supported way to allocate labels more selectively—not proof that objective conflicts have been solved.