用线性映射实现跨模型微调迁移，效果接近完整微调。
Task Matrices: Linear Maps for Cross-Model Finetuning Transfer
- 提出任务矩阵，作为预训练与微调间状态的线性变换。
- 在10个数据集上超越线性探测，接近全模型微调性能。
- 方法高效通用，适合快速适配多领域任务。
可解释性研究提示，大型视觉与语言模型在上下文提示偏置下会学习隐式线性编码。然而，更普遍适应场景中是否存在类似线性表示仍未被证实。本文提出任务矩阵概念，即从基础模型到微调嵌入状态的线性变换。实验表明，在视觉与文本模型及十个不同数据集上，加入任务矩阵的基模型性能优于线性探测，有时接近完全微调水平。结果验证了预训练与微调架构间存在跨层线性编码。此外，我们展示了基于数据的编码近似方法兼具高效性与多领域泛化能力。代码已公开。
原文摘要 · Abstract (English)
Results in interpretability suggest that large vision and language models learn implicit linear encodings when models are biased by in-context prompting. However, the existence of similar linear representations in more general adaptation regimes has not yet been demonstrated. In this work, we develop the concept of a task matrix, a linear transformation from a base to finetuned embedding state. We demonstrate that for vision and text models and ten different datasets, a base model augmented with a task matrix achieves results surpassing linear probes, sometimes approaching finetuned levels. Our results validate the existence of cross-layer linear encodings between pretrained and finetuned architectures. Moreover, we show that a data-based approximation for such encodings is both efficient and generalizable to multiple domains. We make our implementation publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。