用进化算法生成能触发大模型过度拒绝的指令，提升评估与缓解能力。
EVOREFUSE: Evolutionary Prompt Optimization for Evaluation and Mitigation of LLM Over-Refusal to Pseudo-Malicious Instructions
- 通过进化算法迭代优化指令，生成多样且高效的伪恶意请求。
- 新数据集在9个模型上平均拒绝率提升85.34%，响应置信度提高40.03%。
- 适合安全对齐研究者、模型评估人员及希望减少误拒的开发者使用。
大型语言模型（LLMs）常对语义无害的输入指令过度拒绝，源于保守的安全对齐机制，严重影响用户体验。收集此类指令对评估与缓解过度拒绝至关重要，但现有方法如人工创建或指令重写，或缺乏可扩展性，或无法生成足够多样且有效的诱导拒绝提示。为此，我们提出EVOREFUSE，一种基于进化算法的提示优化方法，可持续生成能引发多个LLM一致高置信度拒绝的伪恶意指令。该方法通过突变策略与重组机制，在指令空间中探索更广方向，并迭代优化种子指令以最大化拒绝概率的证据下界。基于此，我们构建两个新数据集：EVOREFUSE-TEST（582条指令），在无安全前缀提示时，跨9个模型平均拒绝触发率比次优基准高出85.34%，词汇多样性提升34.86%，模型响应置信度提高40.03%；EVOREFUSE-ALIGN（3,000条指令+响应），可用于监督与偏好对齐训练。在该数据集上微调的LLAMA3.1-8B-INSTRUCT，相比次优对齐数据训练的模型，过量拒绝减少29.85%，且不损害安全性。分析表明，模型过度拒绝主要因过度关注敏感关键词而忽略上下文。代码与数据集已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) frequently refuse to respond to pseudo-malicious instructions: semantically harmless input queries triggering unnecessary LLM refusals due to conservative safety alignment, significantly impairing user experience. Collecting such instructions is crucial for evaluating and mitigating over-refusals, but existing instruction curation methods, like manual creation or instruction rewriting, either lack scalability or fail to produce sufficiently diverse and effective refusal-inducing prompts. To address these limitations, we introduce EVOREFUSE, a prompt optimization approach that generates diverse pseudo-malicious instructions consistently eliciting confident refusals across LLMs. EVOREFUSE employs an evolutionary algorithm exploring the instruction space in more diverse directions than existing methods via mutation strategies and recombination, and iteratively evolves seed instructions to maximize evidence lower bound on LLM refusal probability. Using EVOREFUSE, we create two novel datasets: EVOREFUSE-TEST, a benchmark of 582 pseudo-malicious instructions that outperforms the next-best benchmark with 85.34% higher average refusal triggering rate across 9 LLMs without a safety-prior system prompt, 34.86% greater lexical diversity, and 40.03% improved LLM response confidence scores; and EVOREFUSE-ALIGN, which provides 3,000 pseudo-malicious instructions with responses for supervised and preference-based alignment training. With supervised fine-tuning on EVOREFUSE-ALIGN, LLAMA3.1-8B-INSTRUCT achieves up to 29.85% fewer over-refusals than models trained on the second-best alignment dataset, without compromising safety. Our analysis with EVOREFUSE-TEST reveals models trigger over-refusals by overly focusing on sensitive keywords while ignoring broader context. Our code and datasets are available at https://github.com/FishT0ucher/EVOREFUSE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。