arXiv:2605.06632cs.LG2026-05

让大模型的微调行为可逆,通过稀疏结构实现可控关闭。

Crafting Reversible SFT Behaviors in Large Language Models

论文配图:Crafting Reversible SFT Behaviors in Large Language Models
图 1 · 摘自论文原文
  • 设计稀疏载体网络,使微调行为集中于特定子网络。
  • 使用软提示触发后可完全逆转微调行为,成功率超90%。
  • 适合关注模型安全与行为可控性的研究人员使用。

监督微调(SFT)会为大语言模型引入新行为,但未对这些行为在模型中的分布施加结构约束。现有方法如电路归因只能事后识别与行为相关的稀疏子网络,但无法证明其因果必要性,限制了推理时对行为的精准控制。本文提出两种新方法:(a) 损失约束双重下降(LCDD),通过联合优化路由掩码与模型权重,在明确的资源预算下构建稀疏且机制必需的“载体”;(b) SFT-Eraser,一种基于激活匹配优化的软提示,用于触发载体通道以逆转微调行为。在多个模型家族上的安全、固定响应和风格类行为测试中,LCDD生成的稀疏载体能有效保留目标行为,且在触发后可实现强逆转。消融实验表明,稀疏结构是可逆性的关键前提:相同触发策略在标准SFT模型上失效,证明结构而非触发设计才是决定因素。结果直接证明所学载体对行为具有因果必要性,为部署模型中系统定位与选择性抑制微调行为提供了新路径。

原文摘要 · Abstract (English)

Supervised fine-tuning (SFT) induces new behaviors in large language models, yet imposes no structural constraint on how these behaviors are distributed within the model. Existing behavior interpretation methods, such as circuit attribution approaches, identify sparse subnetworks correlated with SFT-induced behaviors post-hoc. However, such correlations do not imply *causal necessity*, limiting the ability to selectively control SFT-induced behaviors at inference time. We pursue an alternative by asking: can an SFT-induced behavior be deliberately compressed into a sparse, mechanistically necessary subnetwork, termed a *carrier*, while remaining controllable at inference time without weight modification? We propose (a) **Loss-Constrained Dual Descent (LCDD)**, which constructs such carriers by jointly optimizing routing masks and model weights under an explicit utility budget, and (b) **SFT-Eraser**, a soft prompt optimized via activation matching on extracted carrier channels, to reverse the SFT-induced behavior. Across safety, fixed-response, and style behaviors on multiple model families, LCDD yields sparse carriers that preserve target behaviors while enabling strong reversion when triggered by SFT-Eraser. Ablations further establish that the sparse structure is the key precondition for reversal: the same trigger optimization fails on standard SFT models, confirming that structure rather than trigger design is the operative factor. These results provide direct evidence that the learned carriers are causally necessary for the behaviors, pointing to a new direction for systematically localizing and selectively suppressing SFT-induced behaviors in deployed models.

模型可控性微调压缩行为逆转

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。