arXiv:2603.13359cs.AI2026-03被引 1

通过可调控的拒绝方向,实现对模型拒绝行为的精细控制。

From Refusal Tokens to Refusal Control: Discovering and Steering Category-Specific Refusal Directions

  • 在残差流中发现类别对齐的拒绝方向,用轻量探针提取
  • 构建低秩组合向量,在不重新训练下跨模型迁移
  • 减少对良性提示的过度拒绝,提升有害提示拒绝率

语言模型通常通过微调以安全对齐,拒绝有害提示。一种方法是微调模型生成区分不同拒绝类型的类别拒绝标记。本文利用经过此类标记微调的 Llama 3 8B 模型,在推理阶段实现细粒度拒绝行为的控制,提升了安全性和可靠性。我们发现,拒绝标记微调会在残差流中诱导出可分离、类别对齐的方向,通过轻量探针提取并构建类别可控的引导向量,决定推理时是否朝向或远离拒绝。此外,我们引入一种学习得到的低秩组合,在白化正交基下混合这些类别方向,形成单一可调控的干预机制,有效应对激活空间各向异性。该干预在同架构模型变体间可迁移,无需额外训练。在多个基准测试中,类别引导向量和低秩组合均一致降低了对良性提示的过早拒绝,同时提高了对有害提示的拒绝率,展现出多类别拒绝控制的实用价值。

原文摘要 · Abstract (English)

Language models are commonly fine-tuned for safety alignment to refuse harmful prompts. One approach fine-tunes them to generate categorical refusal tokens that distinguish different refusal types before responding. In this work, we leverage a version of Llama 3 8B fine-tuned with these categorical refusal tokens to enable inference-time control over fine-grained refusal behavior, improving both safety and reliability. We show that refusal token fine-tuning induces separable, category-aligned directions in the residual stream, which we extract and use to construct categorical steering vectors with a lightweight probe that determines whether to steer toward or away from refusal during inference. In addition, we introduce a learned low-rank combination that mixes these category directions in a whitened, orthonormal steering basis, resulting in a single controllable intervention under activation-space anisotropy, and show that this intervention is transferable across same-architecture model variants without additional training. Across benchmarks, both categorical steering vectors and the low-rank combination consistently reduce over-refusals on benign prompts while increasing refusal rates on harmful prompts, highlighting their utility for multi-category refusal control.

模型控制安全对齐拒绝机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。