用模型自生成的难例训练，显著提升安全性和拒答率。
Self-Mined Hardness for Safety Fine-Tuning
- 用模型自身生成结果的有害性评分筛选难例
- 攻击成功率从20%降至3%，拒答率升至94%
- 适合需要高安全性且容忍部分误拒的部署场景
语言模型的安全微调通常依赖人工标注的对抗数据集。本文提出新方法：根据目标模型自身生成结果被判定为有害的频率，评估候选提示的难度，并在最难的提示与模型自身未越狱输出上进行微调。在 Llama-3-8B-Instruct 和 Llama-3.2-3B-Instruct 上，该方法将 WildJailbreak 攻击成功率从 11.5% 和 20.1% 降低至 1%-3%，但将对具有越狱特征的良性提示的拒答率从 14%-22% 提升至 74%-94%。通过以 1:1 比例混合难例与伪装成越狱的良性提示，可将拒答率降至 30%-51%（8B）和 52%-72%（3B），代价为攻击成功率上升 2-6 个百分点。在混合训练中，仅使用最困难的一半样本而非随机一半，可使剩余攻击成功率再降低 35%-50%（约 3 个百分点），两种模型均有效。
原文摘要 · Abstract (English)
Safety fine-tuning of language models typically requires a curated adversarial dataset. We take a different approach: score each candidate prompt's difficulty by how often the target model's own rollouts are judged harmful, then fine-tune on the hardest prompts paired with the model's own non-jailbroken rollouts. On Llama-3-8B-Instruct and Llama-3.2-3B-Instruct, this approach cuts the WildJailbreak attack success rate from 11.5% and 20.1% down to 1-3%, but pushes refusal on jailbreak-shaped benign prompts from 14-22% to 74-94%. Interleaving the same hard prompts 1:1 with adversarially-framed benign prompts (prompts that look like jailbreaks but have benign intent) cuts that refusal back down to 30-51% on 8B and 52-72% on 3B, at a cost of 2-6 percentage points of attack success rate. Within the mixed regime, training on the hardest half of the eligible pool rather than a random half cuts the remaining ASR by 35-50% (about 3 percentage points) on both models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。