为微调后的大模型生成最小干扰的安全补丁,兼顾安全与任务性能。
TRACE: Trajectory-Based Safety Patch Learning for LLM Post-Training Realignment

- 基于轨迹模拟生成有害状态,优化可插拔安全补丁。
- 在六个基准上实现接近100%安全率,性能损失极小。
- 适合需要安全对齐又不愿重训的模型服务提供商。
微调即服务(FTaaS)平台允许用户在定制任务上训练大语言模型(LLM),但此流程可能破坏模型的安全对齐。实际中,服务提供方需在不重新运行完整对齐或破坏已获得的任务效用的前提下恢复模型安全。现有方法采用参数融合技术,在微调后模型参数上添加安全补丁以引导模型远离不安全倾向。然而,此类融合范式受困于任务-安全更新的纠缠问题:下游任务更新与安全补丁常在主导方向上重叠,导致融合强度难以校准。若安全向量过弱,有害成分仍占主导;若过强,则抑制任务相关方向,损害效用。为此,我们转向融合方法的离线补丁学习,寻找在最小干扰任务方向的同时,仍能有效控制不安全行为的补丁。提出TRACE框架,通过(i)模拟有害微调轨迹生成渐进式污染状态,(ii)优化一个可插拔补丁,使其在不同污染状态下恢复安全并保持效用。在六个基准和两个模型上,TRACE持续优于安全-效用前沿。在所有设置下接近100%安全率,且效用与未防御的微调模型相当。
原文摘要 · Abstract (English)
Fine-Tuning-as-a-Service (FTaaS) platforms let users train large language models (LLMs) on customized tasks, but this pipeline could erode models' safety alignment. In practice, service providers need to recover models' safety without re-running full alignment, or destroying the utility gained from customized tasks. A line of existing work refers to model parameter merging, which adds a safety patch on the fine-tuned model parameters to shift the model away from unsafe tendencies. However, this merging-based paradigm is fundamentally bottlenecked by task-safety update entanglement: downstream task updates and the safety patch often overlap in their dominant directions, so the merge strength is intrinsically hard to calibrate. If the safety vector is scaled too weakly, harmful components could still dominate, preventing the model from returning to a safe region; if it is scaled too aggressively, it suppresses task-relevant directions and degrades utility. To solve this problem, we shift the focus of merging-based methods from designing online merging operators to offline patch learning, and seek a safety patch that minimally interferes with task-relevant directions while retaining decisive control over unsafe behaviors. We propose TRACE, a trajectory-based safety patch learning framework that (i) simulates harmful tuning trajectories to generate progressively corrupted states, and (ii) optimizes a plug-in patch to recover safety while maintaining utility across varying corrupted base states. Across six benchmarks and two models, TRACE consistently dominates the safety-utility frontier. TRACE reaches nearly 100% safety on all settings, while maintaining comparable utility to the undefended fine-tuned model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。