让非线性微调模型继承线性微调的特性，实现高效任务组合。
Distilling Linearized Behavior into Non-Linear Fine-Tuning for Effective Task Arithmetic
- 通过激活空间约束，将线性教师模型的知识蒸馏到非线性学生模型中
- 在视觉与语言任务上表现优异，且推理无额外开销
- 适合需要快速模型编辑和部署的场景
任务向量组合已成为编辑预训练模型的有前景范式，支持通过加法实现模型合并、减法实现遗忘。在预训练模型切线空间中进行微调（线性微调）已被证明有效，其生成的任务向量天然解耦且抗干扰。然而，线性化模型在训练中表达能力受限，推理时计算成本更高，限制了实际应用。本文提出在激活空间施加约束，使权重扰动的线性性得以在参数空间中保持。具体而言，我们将曲率正则化的线性化教师模型的隐藏表示蒸馏至通过常规微调训练的非线性学生模型。结果表明，该模型继承了线性模型在任务算术中的关键性质，可有效组合任务向量，在多个视觉与语言基准上表现强劲，且不引入任何推理开销。
原文摘要 · Abstract (English)
Task vector composition has emerged as a promising paradigm for editing pre-trained models, enabling model merging through addition and unlearning through subtraction. Fine-tuning in the tangent space of a pre-trained model (linear fine-tuning) has proven effective, as it produces task vectors that are naturally disentangled and resistant to interference. However, linearized models suffer from limited expressivity during training and incur higher computational costs at inference time, which restrict their practical applicability. In this work, we bridge the gap between linear and standard non-linear fine-tuning. We show that linearity with respect to weight perturbations, a property defined in parameter space, can be enforced through constraints in activation space during training. Concretely, we distill hidden representations from a curvature-regularized linearized teacher into a non-linear student trained via conventional fine-tuning. We find that the resulting model inherits key properties of linearized models for task arithmetic, enabling effective composition of task vectors and achieving strong performance across vision and language benchmarks without incurring any inference-time overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。