筛选良性数据中的异常样本微调，会严重破坏大模型安全性
Benign Samples Matter! Fine-tuning On Outlier Benign Samples Severely Breaks Safety
- 从良性数据中识别出最易导致安全退化的异常样本进行微调
- 仅用100个异常样本就显著降低7种主流大模型的安全对齐
- 现有防护方法大多无效，适合关注模型安全的研究者阅读
近期研究发现，即使在完全良性的数据集上微调大型语言模型（LLMs），也会导致其输出有害性显著上升。基于此，我们通过红队测试进一步深化该威胁：分析并识别出良性数据集中对安全退化贡献最大的样本，仅用这些样本进行微调。我们从异常检测角度出发，提出Self-Inf-N方法以识别并提取此类异常样本。实验表明，在良性数据集中选取的100个由Self-Inf-N检测出的异常样本进行微调，会严重损害大模型的安全对齐。在七种主流大模型上的广泛实验显示，该攻击具有高度跨架构可迁移性，且在实际场景中依然有效。令人担忧的是，大多数现有缓解策略无法抵御此攻击，凸显了亟需更鲁棒的对齐防护机制。代码已公开于https://github.com/GuanZihan/Benign-Samples-Matter。
原文摘要 · Abstract (English)
Recent studies have uncovered a troubling vulnerability in the fine-tuning stage of large language models (LLMs): even fine-tuning on entirely benign datasets can lead to a significant increase in the harmfulness of LLM outputs. Building on this finding, our red teaming study takes this threat one step further by developing a more effective attack. Specifically, we analyze and identify samples within benign datasets that contribute most to safety degradation, then fine-tune LLMs exclusively on these samples. We approach this problem from an outlier detection perspective and propose Self-Inf-N, to detect and extract outliers for fine-tuning. Our findings reveal that fine-tuning LLMs on 100 outlier samples selected by Self-Inf-N in the benign datasets severely compromises LLM safety alignment. Extensive experiments across seven mainstream LLMs demonstrate that our attack exhibits high transferability across different architectures and remains effective in practical scenarios. Alarmingly, our results indicate that most existing mitigation strategies fail to defend against this attack, underscoring the urgent need for more robust alignment safeguards. Codes are available at https://github.com/GuanZihan/Benign-Samples-Matter.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。