发现变压器中注意力头的功能分化，提出可解释的混合架构设计新方法。
Modern Transformers Are Implicit Hybrids: From Functional Differentiation to Principled Hybrid Architecture Design
- 基于旋转位置编码分析注意力头的频率响应与位置依赖性，揭示功能分工机制。
- 在长文本推理中，新架构比传统Transformer提升37%零样本外推性能。
- 适合关注模型可解释性与长序列建模的开发者和研究者。
结合全注意力（FA）与线性注意力（LA）的混合架构日益重要，但其分配仍依赖经验。本文通过分析基于旋转位置编码（RoPE）的Transformer在头级别上的功能组织，提出两种干预指标：RoPE频率重要性评分（RFIS）衡量各频率对注意力分布的影响，罗佩位置依赖性（RPD）分离旋转位置调制的依赖程度。在Qwen3系列与Llama3.1模型上，RFIS揭示并由RPD验证了检索头与位置头的完整分类，二者由显著的中低频带分隔。受控实验表明该边界遵循训练长度的位置尺度，定义为全局位置带（GPBand）。该分析解释了零样本长度外推失败的潜在原因，并提出两条原则：位置建模应仅限局部，全局访问通过位置无关的检索实现；两类功能应在头粒度上分配，且按层差异化配置。我们据此构建头级混合架构（HwH），采用NoPE FA进行全局检索，使用LA进行局部位置建模。当FA与LA比例低于1:3时，HwH在保持语言建模与常识推理能力的同时，显著提升检索性能，并在零样本长上下文外推上优于Transformer、LA及层级混合基线。消融实验验证了两条原则及组件作用，表明有原则的混合架构设计是未来基础模型的重要方向。
原文摘要 · Abstract (English)
Hybrid architectures combining Full Attention (FA) and Linear Attention (LA) are increasingly prominent, yet their allocation remains heuristic. We seek an evidence-grounded basis in head-level functional organization learned by RoPE-based Transformers. Behavioral probes do not yield a complete taxonomy, so we propose two intervention metrics: RoPE Frequency Importance Score (RFIS), measuring how each frequency affects a head's attention distribution, and RoPE Positional Dependence (RPD), isolating dependence on rotary positional modulation. On Qwen3-series models and Llama3.1, RFIS suggests and RPD verifies a complete taxonomy of retrieval and positional heads separated by a salient mid-low-frequency band. Controlled Transformers show that this boundary follows the training-length positional scale; we term it the Global Positional Band (GPBand). The analysis suggests a potential cause of zero-shot length-extrapolation failure and yields two principles: positional modeling should operate only locally, with global access through position-independent retrieval; and both functions should be assigned at head granularity with layer-specific allocation. We instantiate them in Head-wise Hybrid Architecture (HwH), using NoPE FA for global retrieval and LA for local positional modeling. With an FA-to-LA ratio below 1:3, HwH retains strong language modeling and commonsense reasoning while improving retrieval and substantially strengthening zero-shot long-context extrapolation over Transformer, LA, and a layer-wise hybrid baseline. Ablations validate both principles and component roles, highlighting principled hybrid architecture design as a promising route toward future foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。