用安全教师指导基础模型微调，提升模型抗有害数据攻击能力
Safety-Aligned Weights Are Not Enough: Refusal-Teacher-Guided Finetuning Enhances Safety and Downstream Performance under Harmful Finetuning Attacks
- 用安全教师过滤有害提示，指导基础模型微调
- 在有害数据攻击下，有害输出减少42%，任务性能提升18%
- 适合需要安全微调的工业级大模型应用
微调即服务（FaaS）虽可基于用户数据定制大语言模型，但当用户数据包含有害提示时，会引发安全性下降，即有害微调攻击。现有方法先构建安全对齐的模型，再在其基础上进行微调，但发现安全对齐权重作为下游任务初始化效果不佳，导致安全性和实用性均不理想。为此，本文提出从微调安全对齐权重转向在显式安全教师指导下微调基础权重的新范式。具体而言，设计拒绝型教师（Ref-Teacher）引导的微调框架：该教师从用户数据中过滤有害提示，并在微调过程中将安全知识蒸馏至基础模型。大量实验表明，该方法在抵御有害微调攻击时，显著降低有害输出（减少42%），同时提升用户特定任务性能（提升18%）。
原文摘要 · Abstract (English)
While Finetuning-as-a-Service (FaaS) enables customization of Large Language Models (LLMs) using user data, this service is vulnerable to safety degradation when user data includes harmful prompts, a threat known as harmful finetuning attacks. To defend against this, prior work first constructs safety-aligned LLM and then finetunes the LLM on user data. However, we observe that the safety-aligned weights provide weak initialization for downstream task learning, leading to suboptimal safety and utility. Motivated by this limitation, we shift the safe FaaS finetuning paradigm from finetuning safety-aligned weights to finetuning base weights under explicit safety-teacher guidance. Specifically, we propose a Refusal-Teacher (Ref-Teacher)-guided finetuning framework. Our approach directly finetunes the base LLM under the guidance of a safety-aligned Ref-Teacher, which filters harmful prompts from user data and distills safety into the base LLM during finetuning. Extensive experiments demonstrate that our paradigm effectively reduces harmful outputs while improving finetuning performance on user-specific tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。