提出自适应公平对齐框架，让大模型更好兼顾不同群体偏好。
APPA: Adaptive Preference Pluralistic Alignment for Fair Federated RLHF of LLMs
- 根据历史对齐效果动态调整各群体奖励权重，实现公平对齐。
- 在三个模型上使最差群体对齐度提升最高28%，且整体性能优于最小化聚合。
- 无需原始偏好数据，适合隐私敏感的分布式对齐场景。
将大语言模型与多样人类偏好对齐需采用多元对齐策略，即单一模型需同时尊重多个不同群体的价值观。在联邦强化学习从人类反馈（FedRLHF）中，这些群体在不集中偏好数据的情况下对齐共享策略，因此公平奖励聚合至关重要。现有方法存在明显权衡：基于平均的聚合会系统性弱化表现最差群体的对齐，而最小值聚合虽优先保障最差群体，却牺牲整体对齐效果。本文提出APPA（自适应偏好多元对齐）框架，通过历史对齐奖励动态重加权群体级奖励。该方法优先提升对齐不足的群体，而不损害已良好对齐的群体，且无需访问原始偏好数据。集成至基于近端策略优化（PPO）的FedRLHF流程，在GLOBALQA和OQA数据集上，针对三个模型家族（Gemma 2 2B、Llama 3.2 3B、Qwen3 0.6B）进行评估，结果表明APPA在公平性与整体对齐之间取得强权衡，相比平均聚合，最差群体对齐度最高提升28%，同时多数配置下整体对齐优于最小值聚合。
原文摘要 · Abstract (English)
Aligning large language models (LLMs) with diverse human preferences requires pluralistic alignment, where a single model must respect the values of multiple distinct groups simultaneously. In federated reinforcement learning from human feedback (FedRLHF), these groups align a shared policy without centralizing preference data, which makes fair reward aggregation essential. Existing aggregation methods exhibit clear trade offs: average based aggregation systematically under aligns worst performing groups, while min aggregation prioritizes worst group performance at the cost of overall alignment. We propose APPA, an Adaptive Preference Pluralistic Alignment framework that dynamically reweights group level rewards based on historical alignment rewards. Our approach prioritizes under aligned groups without degrading well aligned ones, while requiring no access to raw preference data. Integrated into a proximal policy optimization (PPO) based FedRLHF pipeline and evaluated on GLOBALQA and OQA across three model families (Gemma 2 2B, Llama 3.2 3B, Qwen3 0.6B), APPA achieves strong fairness alignment trade offs, improving worst group alignment by up to 28% over average aggregation while maintaining higher overall alignment than min aggregation across most configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。