通过强化优选响应、拒绝近似优选的差评，提升大模型对齐人类偏好的效率与安全性。
HPS: Hard Preference Sampling for Human Preference Alignment
- 采用单样本蒙特卡洛采样，优先选优并坚决排除劣质响应
- 在HH-RLHF和PKU-Safety数据集上显著提升奖励差距，减少有害内容生成
- 适合追求高安全性和高效训练的对话系统开发者
将大语言模型输出与人类偏好对齐是构建安全可控AI系统的关键。尽管基于Plackett-Luce（PL）和Bradley-Terry（BT）模型的偏好优化方法已展现潜力，但仍存在对有害内容处理不佳、劣质响应利用效率低、以及PL模型计算成本高等问题。为此，我们提出硬偏好采样（HPS）框架，通过训练损失聚焦最优响应并拒绝对比接近的劣质响应，增强模型拒绝能力。结合单样本蒙特卡洛采样策略，有效降低计算开销，同时保持对齐质量。理论上，HPS相比现有PL方法提升样本效率，并最大化优选与劣质响应间的奖励差距，确保更清晰区分。在HH-RLHF与PKU-Safety数据集上的实验表明，其性能在BLEU和奖励得分上与现有方法相当，但显著提升奖励差距，从而大幅减少有害内容生成。
原文摘要 · Abstract (English)
Aligning Large Language Model (LLM) responses with human preferences is vital for building safe and controllable AI systems. While preference optimization methods based on Plackett-Luce (PL) and Bradley-Terry (BT) models have shown promise, they face challenges such as poor handling of harmful content, inefficient use of dispreferred responses, and, specifically for PL, high computational costs. To address these issues, we propose Hard Preference Sampling (HPS), a novel framework for robust and efficient human preference alignment. HPS introduces a training loss that prioritizes the most preferred response while rejecting all dispreferred and harmful ones. It emphasizes "hard" dispreferred responses -- those closely resembling preferred ones -- to enhance the model's rejection capabilities. By leveraging a single-sample Monte Carlo sampling strategy, HPS reduces computational overhead while maintaining alignment quality. Theoretically, HPS improves sample efficiency over existing PL methods and maximizes the reward margin between preferred and dispreferred responses, ensuring clearer distinctions. Experiments on HH-RLHF and PKU-Safety datasets validate HPS's effectiveness, achieving comparable BLEU and reward scores while greatly improving reward margins and thus reducing harmful content generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。