用少量有毒回复训练模型，让大模型更安全地回应恶意提问。
Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs
- 仅需少量模型自动生成的不安全回复作为训练数据
- 引入语义代价与负地球移动距离损失，有效抑制危险输出
- 适合资源有限但需高安全性的大模型部署场景
大语言模型在面对恶意提示时生成不安全内容，是其应用中的重大问题。现有方法通常依赖大量高质量人工标注数据，或使用其他模型生成修正数据，成本高且不可靠。本文提出一种高效安全微调方法，仅需从不安全模型自身获取少量不安全回复即可。通过结合语义代价与负地球移动距离（EMD）损失，引导模型避免生成危险内容，并提出EMD损失的新下界，提升优化效率。实验表明，该方法在性能和数据效率上均优于基线，同时分析了过度对齐对语言能力可能造成的负面影响。
原文摘要 · Abstract (English)
Large Language Models (LLMs) generating unsafe responses to toxic prompts is a significant issue in their applications. While various efforts aim to address this safety concern, previous approaches often demand substantial human data collection or rely on the less dependable option of using another LLM to generate corrective data. In this paper, we aim to take this problem and overcome limitations of requiring significant high-quality human data. Our method requires only a small set of unsafe responses to toxic prompts, easily obtained from the unsafe LLM itself. By employing a semantic cost combined with a negative Earth Mover Distance (EMD) loss, we guide the LLM away from generating unsafe responses. Additionally, we propose a novel lower bound for EMD loss, enabling more efficient optimization. Our results demonstrate superior performance and data efficiency compared to baselines, and we further examine the nuanced effects of over-alignment and potential degradation of language capabilities when using contrastive data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。