用外部锚点解决注意力层复用中的结构矛盾，提升模型效率与性能。
Attention Projection Mixing with Exogenous Anchors
- 将锚点投影移出主网络栈，用可学习系数混合多模态信息
- 动态版本在相同损失下减少50%输入令牌数，下游准确率提升1.5倍
- 适合追求高效训练与轻量化部署的NLP研究者
早期注意力投影的跨层复用能提升优化效率和数据利用率，但带来结构冲突：首层需同时作为深层所有层的稳定锚点和有效计算单元。本文指出这一矛盾限制了内部锚点设计的表现。为此提出ExoFormer，通过在序列层堆栈外学习外部锚点投影来化解该冲突。引入统一归一化混合框架，利用可学习系数（支持逐元素、头级、标量三种粒度）对查询、键、值及门控逻辑进行混合，并证明归一化锚源是稳定复用的关键。ExoFormer变体持续优于内部锚点模型；其动态版本在保持验证损失不变的前提下，仅需原始输入令牌数的1/1.5即可实现1.5倍的下游准确率提升。我们通过‘卸载假说’解释其有效性：外部锚点保留了关键令牌身份，使各层可专注特征转换。代码与模型已开源。
原文摘要 · Abstract (English)
Cross-layer reuse of early attention projections can improve optimization and data efficiency, but it creates a structural conflict: the first layer must simultaneously act as a stable, reusable anchor for all deeper layers and as an effective computational block. We demonstrate that this tension constrains the performance of internal-anchor designs. We propose ExoFormer, which resolves the conflict by learning exogenous anchor projections outside the sequential layer stack. We introduce a unified normalized mixing framework that mixes queries, keys, values, and gate logits using learnable coefficients (exploring coefficient granularities: elementwise, headwise, and scalar), and we show that normalizing anchor sources is key to stable reuse. ExoFormer variants consistently outperform their internal-anchor counterparts, and the dynamic variant yields 1.5x downstream accuracy points while matching validation loss using 1.5x fewer tokens than Gated Attention. We explain this efficacy via an Offloading Hypothesis: external anchors preserve essential token identity, allowing layers to specialize exclusively in feature transformation. We release code and models to facilitate future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。