GradShield通过筛选有害数据，让大模型微调后仍保持安全与实用。
GradShield: Alignment Preserving Finetuning
- 基于隐式危害评分筛选微调数据，提前剔除潜在有害样本。
- 在多种任务中将攻击成功率控制在6%以下，同时保持模型性能。
- 适合需要安全微调的工业级大模型应用者。
大型语言模型在微调过程中面临显著的安全对齐风险，因为模型可能被显式或隐式有害数据所影响。即使看似无害的数据也可能无意中引导模型产生非对齐行为。为此，我们提出GradShield，一种原则性过滤方法，在微调阶段通过计算每个数据点的微调隐式危害评分（FIHS）并采用自适应阈值算法，识别并移除潜在有害数据，从而保护模型对齐。我们在不同有害数据比例下的多个实用微调任务上应用GradShield，使用多种指标评估模型的安全性与实用性。结果表明，GradShield优于所有基线方法，始终将攻击成功率（ASR）维持在6%以下，同时保持良好的任务性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) pose a significant risk of safety misalignment after finetuning, as models can be compromised by both explicitly and implicitly harmful data. Even some seemingly benign data can inadvertently steer a model towards misaligned behaviors. To address this, we introduce GradShield, a principled filtering method that safeguards LLMs during finetuning by identifying and removing harmful data points before they corrupt the model's alignment. It removes potentially harmful data by computing a Finetuning Implicit Harmfulness Score (FIHS) for each data point and employs an adaptive thresholding algorithm. We apply GradShield to multiple utility fine-tuning tasks across varying levels of harmful data and evaluate the safety and utility performance of the resulting LLMs using various metrics. The results show that GradShield outperforms all baseline methods, consistently maintaining an Attack Success Rate (ASR) below $6\%$ while preserving utility performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。