提出高效分组偏好优化算法，让大模型更充分学习多回复对比信息。
GroupDPO: Memory efficient Group-wise Direct Preference Optimization
- 分组训练时解耦反向传播，大幅降低内存占用
- 使用多个回复对比显著提升对齐效果，优于单对训练
- 加入正样本NLL项保障训练稳定性和性能
偏好优化广泛用于将大语言模型与用户偏好对齐。现有方法通常每轮只用一个正例-负例对进行训练，忽略了偏好数据集中通常包含的多个候选回复所提供的额外监督信息。为此，近期工作探索了分组偏好优化，即对同一提示的多个回复进行联合对比，但其实际表现和可扩展性受限于分组耦合目标带来的内存开销。本文提出一种内存高效的分组偏好优化算法，在保留梯度信息的同时，于反向传播阶段解耦样本，显著降低峰值内存使用，支持更大分组规模的可扩展训练。在离线与在线对齐设置下，利用多个回复的对比均持续优于单对训练。此外，对正样本加入负对数似然（NLL）项对于性能提升和训练稳定性至关重要。
原文摘要 · Abstract (English)
Preference optimization is widely used to align Large Language Models (LLMs) with preference feedback. However, most existing methods train on a single positive-negative pair per prompt, discarding additional supervision available in preference datasets that typically contain multiple candidate responses. Motivated by this limitation, recent work explores group-wise preference optimization, which jointly contrasts multiple responses for the same prompt, but its empirical behavior and scalability remain underexplored due to the memory overhead of group-coupled objectives. In this work, we introduce a memory-efficient group-wise preference optimization algorithm that preserves gradients while decoupling samples during backpropagation, substantially reducing peak memory usage, which enables scalable training with larger group sizes. Across both offline and online alignment settings, we show that leveraging multiple responses consistently outperforms single-pair training. Furthermore, incorporating a negative log-likelihood (NLL) term on positive responses is critical for both performance gains and training stability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。