不依赖预训练，实现线性复杂度的长序列建模。
Pointer: Linear-Complexity Long-Range Modeling without Pre-training
- 通过层间指针链构建长距离连接，避免逐对计算。
- 序列长度达2048时仍保持95%以上复制任务准确率。
- 适合对效率敏感且无需预训练的长序列场景。
我们提出Pointer，一种新型架构，在无需预训练的情况下实现长序列建模的线性复杂度 $O(NK)$，同时保持优异性能。与标准注意力机制 $O(N^2)$ 的成对交互不同，该方法采用分层指针链，每层的指针选择依赖前一层的位置，通过指针链建立显式的长距离连接。实验表明，相比标准Transformer，Pointer在长序列上实现2到10倍的速度提升；在2048个词元的距离下，复制任务准确率仍超过95%；且能学习可解释的指针模式，揭示结构化依赖建模能力。在效率基准、长程依赖任务及可解释性分析中，Pointer为无需预训练的高效长序列建模提供了有力替代方案。
原文摘要 · Abstract (English)
We introduce Pointer, a novel architecture that achieves linear $O(NK)$ complexity for long-range sequence modeling while maintaining superior performance without requiring pre-training. Unlike standard attention mechanisms that compute $O(N^2)$ pairwise interactions, our approach uses layer-wise pointer chaining where each layer's pointer selection depends on previous layer's pointer positions, creating explicit long-distance connections through pointer chains. We demonstrate that this architecture achieves $2$--$10\times$ speedup on long sequences compared to standard transformers, maintains $>95\%$ accuracy on copy tasks at distances up to 2048 tokens, and learns interpretable pointer patterns that reveal structured dependency modeling. Our experiments on efficiency benchmarks, long-range dependency tasks, and interpretability analysis show that Pointer offers a compelling alternative to attention mechanisms for scenarios requiring efficient long-range modeling without pre-training dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。