通过剪枝有毒子词提升对话生成安全性与多样性
Toxic Subword Pruning for Dialogue Response Generation on Large Language Models
- 从已训练模型中移除有毒子词，不需重新训练
- 显著降低Llama-3.1-6B的毒性输出，提升对话多样性
- 适合需要快速修复模型毒性的应用开发者
如何防范大语言模型生成有害内容是重要研究方向。现有方法多依赖调整模型权重的安全对齐，但成本高且易引发灾难性遗忘。本文提出一种新颖高效的方法——毒子词剪枝（ToxPrune），从训练好的模型的BPE分词中直接移除有毒子词。与以往认为剪枝会损害机器翻译任务的观点不同，我们发现该方法能有效抑制大模型生成有害内容。实验表明，ToxPrune显著改善了NSFW-3B模型在对话生成中的毒性表现，同时使官方Llama-3.1-6B模型在对话多样性指标上明显提升。大量自动评估和人工评测验证了该方法在修复有毒模型及优化非有毒模型对话生成方面的有效性。
原文摘要 · Abstract (English)
How to defend large language models (LLMs) from generating toxic content is an important research area. Yet, most research focused on various model training techniques to remediate LLMs by updating their weights. A typical related research area is safety alignment. This however is often costly and tedious and can expose the model to even more problems such as catastrophic forgetting if the trainings are not carefully handled by experienced NLP practitioners. We thus propose a simple yet effective and novel algorithm, namely \textbf{Tox}ic Subword \textbf{Prun}ing (ToxPrune) to prune the subword contained by the toxic words from BPE in trained LLMs. In contrast to the previous work that demonstrates pruning BPE tokens as harmful to the task of machine translation, we surprisingly found its usefulness in preventing toxic content from being generated on LLMs. Fortunately, our findings suggest that ToxPrune simultaneously improves the toxic language model NSFW-3B on the task of dialogue response generation obviously. We surprisingly found that ToxPrune can even obviously improve official Llama-3.1-6B in the metric of dialogue diversity. Extensive automatic results and human evaluation indicate that ToxPrune could be helpful for both remediating toxic LLMs and improving non-toxic LLMs on the task of dialogue response generation.\footnote{We plan to release the resources to facilitate future work.}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。