用二叉树结构替代注意力，加速长序列建模。
Wave-Attractor-Tree: A Hierarchical Binary Tree Reduction Architecture for Efficient Sequence Modeling
- 构建分层二叉树，递归使用门控线性单元合并序列
- 训练速度更快，长序列依赖建模准确率更高
- 适合需要快速处理长序列的场景，如基因序列分析
本文提出一种基于分层二叉树的序列压缩架构，取代标准自注意力机制。核心思想是通过递归的门控线性单元（GLU）合并操作，在保持 O(n) 总合并次数、O(log n) 并行深度、O(n d²) 总计算量和 O(n) 空间复杂度的前提下，实现高效序列建模。实验表明，该模型在长程结构依赖任务中显著优于标准 Transformer，尤其在需要层次归纳偏置的场景下表现突出。
原文摘要 · Abstract (English)
Work introduces a hierarchical binary tree-based reduction that replaces standard self-attention. The core idea is to use a recursive Gated Linear Unit merge operation, achieving O(n) total merge operations O(log n) parallel depth O(n d^2) total work and O(n) space complexity. In these experiments, the model significantly outperforms standard Transformers in both convergence speed and accuracy on long-range structural dependencies, specifically where hierarchical inductive bias is critical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。