Who Gets to Shape AI? The Case for Distributed Alignment

Human feedback helps turn a language model into an assistant that follows preferences about useful, appropriate responses. But where should that feedback be processed, and who should control it? A preprint by Zhanhong Jiang, Qisai Liu and Soumik Sarkar at Iowa State University argues that these architectural questions belong at the center of AI alignment research.

Their paper, Vision Paper: Why Distributed LLM Post-Training and Decentralized Alignment Are Now Essential, was submitted to arXiv on November 4, 2026. It is principally a vision and research-agenda paper, supported by a small proof-of-concept experiment—not a demonstration that distributed alignment is ready to replace centralized systems.

Keeping judgments local

The authors challenge what they call the centralization assumption: that human judgments about model outputs should be collected and processed under a model developer’s control. They organize their argument around privacy and data sovereignty, representation of diverse preferences, and robustness and scalability.

A clinician’s assessment of an answer may be entangled with protected health information. A legal professional’s judgment may reflect privileged deliberation. Keeping such material within its originating institution could reduce the need to transfer sensitive records. The paper also acknowledges that legal compliance depends on implementation and jurisdiction, including consent, de-identification and data-processing agreements; centralized training is not automatically unlawful.

The representation argument concerns who can participate. The authors contend that centralized annotation pipelines overrepresent English-language, Western and high-income perspectives. Distributed training could let institutions and communities contribute locally generated preferences without surrendering their underlying data. That is a proposed route toward broader participation, not an experimentally established equity improvement in this study.

Two ways to collaborate

The paper develops distributed versions of direct preference optimization, or DPO. This method trains a model using pairs of preferred and dispreferred responses, comparing their probabilities against a reference model.

In federated DPO, institutions train locally and send model updates to a coordinating server, which combines them. In decentralized DPO, there is no aggregation server: participating nodes exchange information with neighbors in a network. Both approaches retain raw preference data locally, while differential privacy and secure aggregation are discussed as additional protections.

A central difficulty is reference drift. Because DPO depends on a reference model, participants using different references can optimize different implicit rewards even when their current policies are identical. Combining their updates can therefore mix inconsistent objectives.

The proposed federated remedy is Synchronized Reference Update: periodically broadcasting the current global policy so every client adopts the same new reference. For decentralized training, the authors propose periodic reference resets based on neighboring policies. These are proposed coordination mechanisms, not universally validated fixes.

The experiment complicates the pitch

The appendix tests five simulated clients using a roughly 16-million-parameter GPT-2-style model built from scratch, without pretrained weights. Training updates only low-rank adapters, a parameter-efficient technique known as LoRA. The study uses 120 preference pairs from the HH-RLHF and Stanford Human Preferences datasets, with 60 pairs held out for evaluation. Clients receive different mixtures of the two sources.

After 50 communication rounds, centralized DPO achieves the lowest reported loss, approximately 0.42. Federated DPO with synchronized references reaches about 0.62, compared with roughly 0.68–0.70 for federated training without synchronization.

The decentralized results are less straightforward. Training without neighborhood reference resets achieves the best distributed loss, about 0.55. Adding neighborhood consensus every four rounds instead leaves loss around 0.68–0.70. The authors interpret this as sensitivity to the reset schedule: repeatedly changing the reference can disrupt learning before neighboring policies have converged.

That outcome matters because it qualifies the paper’s broad advocacy. Coordination helped in the federated experiment, but the tested decentralized coordination schedule did not. Choosing when to synchronize remains an open research problem.

Privacy, disagreement and accountability

The experiments also explore added privacy noise. Reported preference win rates fluctuate rather than improving smoothly as noise decreases, and all distributed conditions remain below the centralized baseline. The authors emphasize the small dataset and the variability of these results. Their win-rate measure reflects a model-based preference margin, not direct human judgment of response quality.

Other unresolved problems are institutional as well as mathematical. Genuine cultural or professional disagreements can produce updates that resemble malicious manipulation. The paper does not offer a complete solution for distinguishing the two. Serverless collaboration also raises questions about auditing, excluding malicious participants and assigning responsibility.

The authors call for convergence theory, adaptive synchronization, personalized alignment and benchmarks that report worst-client performance rather than averages alone. Their strongest supported message is a research priority: investigate how locally held preferences can shape shared models, while treating privacy, diversity, safety and accountability as problems still requiring evidence and design.