通过递归注意力提升模型捕捉复杂关系能力，性能更优且参数几乎不增。
Nexus: Higher-Order Attention Mechanisms in Transformers
- 用嵌套注意力动态优化查询与键向量，增强上下文建模。
- 在多个基准上超越标准Transformer，突破线性瓶颈限制。
- 参数高效设计，递归层间共享权重，额外参数仅常数级增长。
Transformer在多个领域取得显著成功，依赖自注意力机制捕捉依赖关系。然而，标准的一阶注意力机制常受限于低秩瓶颈，难以在单层内捕捉复杂的多跳关系。本文提出Nexus，一种新型架构，通过递归框架增强表征能力。不同于传统方法中静态的线性投影查询与键，Nexus利用嵌套自注意力机制动态优化这些表示。具体而言，查询和键向量本身是内部注意力循环的输出，使标记能在最终注意力计算前聚合全局上下文并建模高阶相关性。我们在递归步骤间采用参数高效的权重共享策略，确保增强表达力仅带来$/mathcal{O}(1)$额外参数。理论分析表明，该方法打破了标准注意力的线性瓶颈。实验上，Nexus在多个基准测试中优于标准Transformer。
原文摘要 · Abstract (English)
Transformers have achieved significant success across various domains, relying on self-attention to capture dependencies. However, the standard first-order attention mechanism is often limited by a low-rank bottleneck, struggling to capture intricate, multi-hop relationships within a single layer. In this paper, we propose the Nexus, a novel architecture designed to enhance representational power through a recursive framework. Unlike standard approaches that use static linear projections for Queries and Keys, Nexus dynamically refines these representations via nested self-attention mechanisms. Specifically, the Query and Key vectors are themselves outputs of inner attention loops, allowing tokens to aggregate global context and model high-order correlations \textit{prior} to the final attention computation. We enforce a parameter-efficient weight-sharing strategy across recursive steps, ensuring that this enhanced expressivity incurs $\mathcal{O}(1)$ additional parameters. We provide theoretical analysis demonstrating that our method breaks the linear bottleneck of standard attention. Empirically, Nexus outperforms standard Transformers on multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。