新方法让语言模型更少生成不良内容，同时不牺牲整体表现。
Reducing the Probability of Undesirable Outputs in Language Models Using Probabilistic Inference
- 通过学习引导采样低分输出并降低其概率来改进训练。
- 在保持平均收益的同时，显著降低不良输出概率。
- 适合对安全性要求高的实际应用，如客服或内容生成。
强化学习（RL）已成为对齐语言模型与人类偏好或提升奖励函数所定义理想输出的主要方法。标准RL优化平均奖励，而专门降低不良输出概率的方法通常会损害整体性能。为此，我们提出RePULSe，一种在标准RL损失基础上增加额外损失的新训练方法：利用学习到的提议策略引导采样低奖励输出，并降低这些输出的概率。实验表明，与标准RL对齐方法及其它替代方案相比，RePULSe在期望奖励与不良输出概率之间实现了更优权衡，且具备更强的对抗鲁棒性。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a predominant technique to align language models (LMs) with human preferences or promote outputs which are deemed to be desirable by a given reward function. Standard RL approaches optimize average reward, while methods explicitly focused on reducing the probability of undesired outputs typically come at a cost to average-case performance. To improve this tradeoff, we introduce RePULSe, a new training method that augments the standard RL loss with an additional loss that uses learned proposals to guide sampling low-reward outputs, and then reduces those outputs' probability. We run experiments demonstrating that RePULSe produces a better tradeoff of expected reward versus the probability of undesired outputs and is more adversarially robust, compared to standard RL alignment approaches and alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。