只在必要时干预生成,提升安全同时保持原模型质量。
Selective Safety Steering via Value-Filtered Decoding

- 用价值过滤机制选择性干预,避免对本就安全的输出过度修改。
- 通过单个阈值控制误干预概率,实现安全与原生质量的平衡。
- 适合追求高安全且不希望模型风格被破坏的应用场景。
尽管大语言模型(LLMs)训练时已对齐人类价值观,其生成内容仍可能违反安全约束。现有解码阶段的安全干预方法常对本就安全的输出进行无谓修改,导致模型原本的有用性、流畅性、风格和连贯性受损。本文提出一种新的测试时安全引导方法——基于价值筛选的解码策略,仅在必要时干预生成,同时提供误干预概率的显式上界。单一阈值超参数可调节该上界,使用户在更高误干预率与更好输出安全性之间灵活权衡。在多个数据集和实验中,该方法均优于现有基线,在安全性和与基础模型相似性之间实现了更优平衡。
原文摘要 · Abstract (English)
While large language models (LLMs) are trained to align with human values, their generations may still violate safety constraints. A growing line of work addresses this problem by modifying the model's sampling policy at decoding time using a safety reward. However, existing decoding-time steering methods often intervene unnecessarily, modifying generations that would have been safe under the base model. Such unnecessary interventions are undesirable, as they can distort key properties of the base model such as helpfulness, fluency, style, and coherence. We propose a new test-time steering method designed to reduce such unnecessary interventions while improving the safety of unsafe responses. Our approach filters tokens using a value-based safety criterion and provides an explicit bound on the probability of false interventions. A single threshold hyperparameter controls this bound, allowing practitioners to trade off higher rates of unnecessary intervention for better output safety. Across multiple datasets and experiments, we show that our value-filtered decoding method outperforms existing baselines, achieving better trade-offs between safety, helpfulness, and similarity to the base model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。