研究小模型在微调中特征的演化与融合，揭示跨领域迁移规律。
Tracking Universal Features Through Fine-Tuning and Model Merging
- 基于单层Transformer模型，通过微调和球面插值融合不同领域文本
- 发现部分特征在微调后仍保持稳定，而新任务引入了特定于领域的特征
- 适用于对模型可解释性、迁移学习机制感兴趣的科研人员
我们研究了在不同文本领域上微调时，模型特征如何出现、消失与持续存在。具体而言，从一个在BabyLM语料库及The Stack中的Python代码组合上训练的单层Transformer语言模型出发，分别将其适配到TinyStories和Lua编程语言两个新领域，并使用球面线性插值（spherical linear interpolation）对这两个模型进行合并。本研究旨在通过小型模型和稀疏自编码器，深入理解典型迁移学习场景下特征的稳定性和演变过程。
原文摘要 · Abstract (English)
We study how features emerge, disappear, and persist across models fine-tuned on different domains of text. More specifically, we start from a base one-layer Transformer language model that is trained on a combination of the BabyLM corpus, and a collection of Python code from The Stack. This base model is adapted to two new domains of text: TinyStories, and the Lua programming language, respectively; and then these two models are merged using these two models using spherical linear interpolation. Our exploration aims to provide deeper insights into the stability and transformation of features across typical transfer-learning scenarios using small-scale models and sparse auto-encoders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。