提出高效处理高维长序列的新架构ENA，兼顾全局与局部建模。
ENA: Efficient N-dimensional Attention
- 结合线性循环与分块滑动窗口注意力，实现高效高维建模。
- 在长序列任务中性能优于传统Transformer，计算开销更低。
- 适合处理超长高维数据，如视频、医学影像等场景。
长序列高阶数据的高效建模需要超越Transformer的架构。本文研究将原本为语言建模设计的线性递归模型扩展至高阶数据（1D到ND）的两个关键方面：扫描策略和注意力混合架构。实验表明，扫描策略带来的收益有限，而注意力混合模型表现优异。进一步评估不同注意力类型后发现，分块高阶滑动窗口注意力（SWA）在理论上和实践中均具高效性。由此提出的线性递归与高阶SWA融合的混合架构称为高效多维注意力（ENA）。实验验证了其有效性。ENA的核心思想是：线性递归将全局信息压缩至状态，而SWA通过严格局部建模进行补充。两者结合形成简洁且实用的超长高维数据建模方案。
原文摘要 · Abstract (English)
Efficient modeling of long sequences of high-order data requires a more efficient architecture than Transformer. In this paper, we investigate two key aspects of extending linear recurrent models, especially those originally designed for language modeling, to high-order data (1D to ND): scanning strategies and attention-hybrid architectures. Empirical results suggest that scanning provides limited benefits, while attention-hybrid models yield promising results. Focusing on the latter, we further evaluate types of attention and find that tiled high-order sliding window attention (SWA) is efficient in both theory and practice. We term the resulting hybrid architecture of linear recurrence and high-order SWA as Efficient N-dimensional Attention (ENA). We then conduct several experiments to demonstrate its effectiveness. The intuition behind ENA is that linear recurrence compresses global information into a state, while SWA complements it by enforcing strict local modeling. Together, they form a simple framework that offers a promising and practical solution for ultra-long high-order data modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。