用双层优化筛选安全数据，让大模型微调更安全可靠
SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selection
- 通过双层优化学习数据排序器，优先选择安全高质量数据
- 在Llama-3-8b-Instruct上提升8.5%胜率，Merlinite-7b上提升9.7%
- 适合需要安全微调的场景，如医疗、金融等敏感领域
在特定任务数据上微调以提升下游性能是利用大语言模型的关键步骤。然而，已有研究发现，在对抗样本甚至良性数据上微调会严重损害模型原有的对齐与安全能力。本文提出SEAL框架，通过双层优化学习数据排序器，对安全且高质量的数据进行高排名，对不安全或低质量的数据进行低排名。使用SEAL训练的模型在多个基准上表现更优，相较于随机选择，在Llama-3-8b-Instruct和Merlinite-7b模型上的胜率分别提升了8.5%和9.7%。代码已开源于GitHub：https://github.com/hanshen95/SEAL。
原文摘要 · Abstract (English)
Fine-tuning on task-specific data to boost downstream performance is a crucial step for leveraging Large Language Models (LLMs). However, previous studies have demonstrated that fine-tuning the models on several adversarial samples or even benign data can greatly comprise the model's pre-equipped alignment and safety capabilities. In this work, we propose SEAL, a novel framework to enhance safety in LLM fine-tuning. SEAL learns a data ranker based on the bilevel optimization to up rank the safe and high-quality fine-tuning data and down rank the unsafe or low-quality ones. Models trained with SEAL demonstrate superior quality over multiple baselines, with 8.5% and 9.7% win rate increase compared to random selection respectively on Llama-3-8b-Instruct and Merlinite-7b models. Our code is available on github https://github.com/hanshen95/SEAL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。