让生成模型避开安全检测边界，提升内容安全与质量
Don't Walk the Line: Boundary Guidance for Filtered Generation
- 用强化学习引导生成远离安全分类器的决策边界
- 在越狱、模糊和长文本提示下显著降低误判率
- 适合需要高安全性和可用性的生成系统开发者
生成模型常搭配安全分类器以过滤有害或不期望的输出。常见做法是微调生成器以降低被过滤的概率，但此方法常导致模型生成靠近分类器决策边界的样本，从而增加误报和漏报。本文提出边界引导（Boundary Guidance），一种基于强化学习的微调方法，可显式引导生成远离分类器的决策边界。在越狱、模糊和长上下文提示的基准测试中，该方法提升了输出的安全性与实用性，经大模型作为评判者评估验证。跨模型规模与奖励设计的全面消融实验表明该方法具有强鲁棒性。
原文摘要 · Abstract (English)
Generative models are increasingly paired with safety classifiers that filter harmful or undesirable outputs. A common strategy is to fine-tune the generator to reduce the probability of being filtered, but this can be suboptimal: it often pushes the model toward producing samples near the classifier's decision boundary, increasing both false positives and false negatives. We propose Boundary Guidance, a reinforcement learning fine-tuning method that explicitly steers generation away from the classifier's margin. On a benchmark of jailbreak, ambiguous, and longcontext prompts, Boundary Guidance improves both the safety and the utility of outputs, as judged by LLM-as-a-Judge evaluations. Comprehensive ablations across model scales and reward designs demonstrate the robustness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。