提出自适应对比解码,减少大模型误拒无害请求。
Please refuse to answer me! Mitigating Over-Refusal in Large Language Models via Adaptive Contrastive Decoding

- 通过对比极端安全提示下的输出分布,动态调整拒绝概率。
- 在五个数据集上平均降低误拒率10.35%,恶意请求拒答率提升0.13%。
- 无需训练、不依赖模型,适合部署于各类安全对齐大模型。
安全对齐的大语言模型常因过量拒绝问题,对无害查询产生拒绝响应。现有方法难以同时保持对无害请求的低拒绝率与对恶意请求的高拒绝率。本文分析不同安全级别系统提示对模型拒绝行为的影响,发现当模型出现过量拒绝时,非拒绝词仍存在于候选词列表中,但模型系统性地未选择它们。基于此,提出无需训练、模型无关的自适应对比解码(AdaCD)方法:首先,比较有无极端安全提示下的输出分布以优化拒绝词分布;其次,引入自适应对比解码策略,动态调整是否引入拒绝分布,从而自适应提升拒绝或非拒绝词的选择概率。在五个基准数据集上的实验表明,平均而言,AdaCD将过量拒绝查询的拒绝率降低10.35%,同时使恶意查询的拒绝率提高0.13%。代码已公开于https://github.com/OutdoorManofML/AdaCD。
原文摘要 · Abstract (English)
Safety-aligned large language models (LLMs) often generate refusal responses to harmless queries due to the over-refusal problem. However, existing methods for mitigating over-refusal cannot maintain a low refusal ratio for harmless queries while keeping a high refusal ratio for malicious ones. In this paper, we analyze how system prompts with varying safety levels affect LLM refusal behaviors when facing over-refusal queries. A key observation is that, when LLMs suffer from the over-refusal issue, non-refusal tokens remain present in the next-token candidate list, but the model systematically fails to select them, despite the generation of refusal tokens. Based on this observation, we propose a training-free and model-agnostic approach, Adaptive Contrastive Decoding (AdaCD), to mitigate over-refusal while maintaining LLM safety. First, AdaCD compares the output distributions of the LLM with or without an extreme safety system prompt to refine the refusal token distribution. Second, we introduce an adaptive contrastive decoding strategy that dynamically incorporates or removes the refusal token distribution, adaptively boosting the probability of selecting refusal or non-refusal tokens. Experimental results on five benchmark datasets show that, on average, AdaCD reduces the refusal ratio for over-refusal queries by 10.35%, yet still increases the refusal ratio for malicious queries by 0.13%. Code is available at https://github.com/OutdoorManofML/AdaCD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。