用简单映射在模型间迁移线性特征，省下50%训练成本。
Transferring Linear Features Across Language Models With Model Stitching
- 通过仿射变换映射残差流，实现模型间特征高效迁移。
- 小模型训练的稀疏自编码器迁移到大模型，节省50%计算量。
- 语义与结构特征迁移效果不同，适合优化大模型特征学习。
本文表明，语言模型残差流之间的仿射映射是一种低成本但高效的特征迁移方法。我们将其用于在不同规模模型间迁移稀疏自编码器（SAEs）权重，以比较其表征能力。结果发现，小模型与大模型学习到的表示空间相似，这促使我们在小模型上训练昂贵组件如SAEs，再迁移到大模型，显著降低计算开销。具体而言，使用小模型到大模型的转移初始化，可使大模型上SAE的训练成本降低50%。此外，转移后的探测器和导向向量能有效恢复真实性能。深入分析显示，语义与结构特征的迁移表现明显不同，而特定功能类特征则被精确映射。整体表明小大模型在线性表示空间上存在相似性与差异性，并提出一种提升SAE训练效率的新方法。
原文摘要 · Abstract (English)
In this work, we demonstrate that affine mappings between residual streams of language models is a cheap way to effectively transfer represented features between models. We apply this technique to transfer the weights of Sparse Autoencoders (SAEs) between models of different sizes to compare their representations. We find that small and large models learn similar representation spaces, which motivates training expensive components like SAEs on a smaller model and transferring to a larger model at a FLOPs savings. In particular, using a small-to-large transferred SAE as initialization can lead to 50% cheaper training runs when training SAEs on larger models. Next, we show that transferred probes and steering vectors can effectively recover ground truth performance. Finally, we dive deeper into feature-level transferability, finding that semantic and structural features transfer noticeably differently while specific classes of functional features have their roles faithfully mapped. Overall, our findings illustrate similarities and differences in the linear representation spaces of small and large models and demonstrate a method for improving the training efficiency of SAEs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。