提出新方法提升大模型安全对齐，避免部分响应不安全。
Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization
- 用每条提示的严格安全约束替代期望安全约束。
- 在多数提示上实现更优的安全性，避免安全补偿问题。
- 适合关注大模型安全对齐的研究者和开发者。
在大语言模型（LLM）对齐中，如何平衡有用性与安全性（无害性）是一项关键挑战。现有方法通常将二者解耦，分别训练有用性和安全性偏好模型，并将安全性视为约束马尔可夫决策过程（CMDP）中的约束。本文指出，广泛采用的期望安全约束存在“安全补偿”问题：虽然整体满足约束，但个别提示可能在安全与有用间权衡，导致某些响应过于保守，而另一些仍不安全。为此，本文提出修正策略优化（RePO），将期望安全约束替换为针对每条提示的严格安全约束。RePO 的核心是基于修正策略梯度的更新机制，对每条提示的安全违规进行惩罚，从而显著提升几乎所有提示下的安全性。实验表明，RePO 在多个基准上优于强基线方法，显著增强了大模型的安全对齐效果。
原文摘要 · Abstract (English)
Balancing helpfulness and safety (harmlessness) is a critical challenge in aligning large language models (LLMs). Current approaches often decouple these two objectives, training separate preference models for helpfulness and safety, while framing safety as a constraint within a constrained Markov Decision Process (CMDP) framework. This paper identifies a potential issue when using the widely adopted expected safety constraints for LLM safety alignment, termed "safety compensation", where the constraints are satisfied on expectation, but individual prompts may trade off safety, resulting in some responses being overly restrictive while others remain unsafe. To address this issue, we propose Rectified Policy Optimization (RePO), which replaces the expected safety constraint with critical safety constraints imposed on every prompt. At the core of RePO is a policy update mechanism driven by rectified policy gradients, which penalizes the strict safety violation of every prompt, thereby enhancing safety across nearly all prompts. Our experiments demonstrate that RePO outperforms strong baseline methods and significantly enhances LLM safety alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。