发现不同语言模型的注意力模式有共性，可无训练迁移提升性能。
Order-Level Attention Similarity Across Language Models: A Latent Commonality
- 基于注意力展开的阶次分解，提取跨模型共性特征
- 相同阶次的注意力相似度高，且与句法知识隐含关联
- 无需训练即可跨模型适配，适合模型迁移场景
本文探讨了一个此前被忽视的重要问题：不同语言模型中的上下文聚合模式是否存在共性？尽管已有研究关注单个模型或注意力头的上下文聚合，但缺乏对多个模型间共性的系统分析。本文提出一种基于注意力展开阶次分解的有序注意力（OLA），发现同一阶次下不同语言模型的OLA具有显著相似性，并揭示了OLA与句法知识之间的隐含映射关系。基于此，我们设计了无需训练的跨模型适配器TOA，将OLA作为统一句法特征输入，利用其跨模型一致性实现对未见模型的泛化。大量实验表明，该方法能有效提升未见模型的性能。代码已开源：https://github.com/jinglin-liang/OLAS。
原文摘要 · Abstract (English)
In this paper, we explore an important yet previously neglected question: Do context aggregation patterns across Language Models (LMs) share commonalities? While some works have investigated context aggregation or attention weights in LMs, they typically focus on individual models or attention heads, lacking a systematic analysis across multiple LMs to explore their commonalities. In contrast, we focus on the commonalities among LMs, which can deepen our understanding of LMs and even facilitate cross-model knowledge transfer. In this work, we introduce the Order-Level Attention (OLA) derived from the order-wise decomposition of Attention Rollout and reveal that the OLA at the same order across LMs exhibits significant similarities. Furthermore, we discover an implicit mapping between OLA and syntactic knowledge. Based on these two findings, we propose the Transferable OLA Adapter (TOA), a training-free cross-LM adapter transfer method. Specifically, we treat the OLA as a unified syntactic feature representation and train an adapter that takes OLA as input. Due to the similarities in OLA across LMs, the adapter generalizes to unseen LMs without requiring any parameter updates. Extensive experiments demonstrate that TOA's cross-LM generalization effectively enhances the performance of unseen LMs. Code is available at https://github.com/jinglin-liang/OLAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。