无需微调即可跨模型迁移任务知识,性能接近直接微调。
Bilinear Coordinate Alignment for Training-Free Task-Vector Transfer

- 将任务向量视为输入激活与输出梯度的双线性累积,构建双空间对齐框架。
- 在视觉与语言任务上,相比现有方法提升显著,且在宽深不一模型间表现稳定。
- 仅需一次前向反向传播即可完成对齐,无需参数更新,适合快速部署。
微调大模型是将通用表征适配到特定任务的主流方法。但当新版本预训练模型发布时,原有微调所得的知识无法直接复用,因它依赖原始模型的参数化,需重新微调,成本高昂。为解决此问题,近期工作采用任务向量(即微调后模型与基模型的参数差)实现跨模型知识迁移。现有方法通过匹配激活或梯度来桥接不同模型,但性能仍远低于直接微调,表明局部对应关系不足。本文重新审视任务向量的生成机制,发现其可表示为输入侧激活与输出侧梯度的累积双线性交互。基于此,我们将任务向量迁移建模为双空间对齐问题,提出BiCo框架,通过单次前向-反向传播在小校准集上估计正交Procrustes映射,无需参数更新。在广泛计算机视觉与自然语言处理基准测试中,BiCo在宽度、深度及预训练配置不同的模型间一致优于现有方法。
原文摘要 · Abstract (English)
Fine-tuning large-scale pre-trained models is a recent prevalent paradigm for adapting general representations to specialized tasks. However, when a new version of a pre-trained model becomes available, expertise acquired through fine-tuning cannot be directly reused because it is tied to the parameterization of the original model, requiring another costly fine-tuning. To address this inefficiency, recent work uses task vectors, defined as the parameter difference between a fine-tuned model and its base model, to transfer expertise across models. While existing methods bridge disparate models by matching activations or gradients, a significant performance gap remains relative to direct fine-tuning, suggesting that these partial correspondences are insufficient. In this work, instead of viewing a task vector merely as a parameter offset, we revisit the formation of task vectors and show that they can be derived as accumulated bilinear interactions between input-side activations and output-side gradients. Motivated by this observation, we formulate task-vector transfer as a dual-space alignment problem and propose BiCo, a training-free framework for transferring task vectors through Bilinear Coordinate alignment. BiCo estimates orthogonal Procrustes mappings in both spaces using a single forward-backward pass on a small calibration set, without any parameter update. Across extensive computer vision and natural language processing benchmarks, BiCo consistently outperforms existing transfer methods across models that differ in width, depth, and pre-training configuration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。