用分组相对优化稳定大模型偏见对齐，兼顾探索与训练稳定性。
BiasGRPO: Stabilizing Bias Mitigation in High-Variance Reward Landscapes via Group-Relative Policy Optimization
- 以分组相对奖励替代价值函数，降低高方差偏见奖励的不稳定性。
- 在多个基准上优于DPO和PPO，实现更稳定且高效的偏见缓解。
- 开源高效偏见评分模型，可无缝接入多目标强化学习对齐流程。
大语言模型中的社会偏见缓解面临独特对齐挑战：由于偏见无单一真实标准，导致奖励空间主观且方差高。现有基于偏好微调的方法存在明显权衡——直接偏好优化（DPO）受限于离线训练缺乏探索性，而近端策略优化（PPO）因批评者估计不可靠易引发训练不稳定。本文提出BiasGRPO，采用分组相对策略优化（GRPO），通过在一组采样生成结果间归一化奖励来稳定对齐过程。该方法以分组相对基线替代值函数，在保持在线训练探索优势的同时显著降低不稳定性。实验表明，BiasGRPO在多个基准上优于DPO和PPO。为适配GRPO，我们合成扩展了覆盖多领域、多情境的数据集，并构建并发布一个高效偏见评分模型，其计算成本低、避免知识退化，可直接集成至多目标强化学习人类反馈（RLHF）流程中。
原文摘要 · Abstract (English)
Mitigating social bias in Large Language Models (LLMs) presents a distinct alignment challenge: unlike verifiable tasks, bias lacks a single ground truth, creating a high-variance, subjective reward landscape. Previous preference-based fine-tuning methods have major trade-offs: Direct Preference Optimization (DPO) is limited by the lack of exploration inherent in offline training, while Proximal Policy Optimization (PPO) can lead to training instability due to potentially unreliable critic estimates. In this paper, we propose BiasGRPO, a framework using Group Relative Policy Optimization (GRPO) to stabilize alignment by normalizing rewards across a group of sampled completions. By substituting the value function with a group-relative baseline, our approach reduces instability while maintaining the exploration benefits of online training. We find that BiasGRPO outperforms DPO and PPO across multiple benchmarks, indicating its effectiveness. To adapt GRPO, we synthetically extend a dataset spanning multiple domains and contexts. We also create and release a custom bias reward model that effectively guides generation while being highly compute-efficient and avoiding knowledge degradation, providing a valuable resource that can be seamlessly integrated into multi-objective RLHF pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。