调参不当会破坏大模型安全，正确设置可大幅降低有害回应
Rethinking Safety in LLM Fine-tuning: An Optimization Perspective
- 通过优化超参数选择，避免安全性能下降
- 有害回复率从16%降至约5%，同时保持模型性能
- 提出EMA动量方法，无需额外安全数据
微调语言模型常被认为会不可避免地损害其安全性，即对有害请求的拒绝能力，即使使用无害数据集也是如此，因此需额外安全措施。我们通过系统测试挑战这一观点，发现安全问题多由不良优化选择导致，而非固有权衡。通过合理选择学习率、批量大小和梯度步数等关键超参数，我们使有害响应率（关键词匹配）从16%降至约5%，同时保持模型效用。基于此，我们提出一种参数空间中的指数移动平均（EMA）动量技术，通过构建稳定优化路径，保留预训练模型的安全特性。在Llama系列模型上，针对Dolly、Alpaca、ORCA等多个数据集的实验表明，通过合理配置可基本避免微调中的安全问题，优于依赖额外安全数据的现有方法，为模型适配提供实用指南。
原文摘要 · Abstract (English)
Fine-tuning language models is commonly believed to inevitably harm their safety, i.e., refusing to respond to harmful user requests, even when using harmless datasets, thus requiring additional safety measures. We challenge this belief through systematic testing, showing that poor optimization choices, rather than inherent trade-offs, often cause safety problems, measured as harmful responses to adversarial prompts. By properly selecting key training hyper-parameters, e.g., learning rate, batch size, and gradient steps, we reduce unsafe model responses from 16\% to approximately 5\%, as measured by keyword matching, while maintaining utility performance. Based on this observation, we propose a simple exponential moving average (EMA) momentum technique in parameter space that preserves safety performance by creating a stable optimization path and retains the original pre-trained model's safety properties. Our experiments on the Llama families across multiple datasets (Dolly, Alpaca, ORCA) demonstrate that safety problems during fine-tuning can largely be avoided without specialized interventions, outperforming existing approaches that require additional safety data while offering practical guidelines for maintaining both model performance and safety during adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。