用递归稀疏结构提升长文本推理的效率与扩展性
ReSSFormer: A Recursive Sparse Structured Transformer for Scalable and Long-Context Reasoning
- 用递归机制替代层层堆叠，实现深度受限的迭代推理
- 通过专家级稀疏注意力聚焦关键信息，减少计算量
- 无需位置编码，直接从内容学习令牌拓扑结构
尽管Transformer在多个领域表现出卓越的可扩展性，但在长上下文推理、计算效率和结构泛化方面仍面临挑战，主要源于固定的层堆叠、密集注意力和对位置编码的依赖。本文提出ReSSFormer，一种递归稀疏结构Transformer，集成三项互补创新：用于有界深度迭代推理的循环推理与记忆单元（R2MU），用于高效聚焦上下文选择的自适应稀疏注意力模块（ASAM），以及实现无位置编码结构归纳的自组织编码器结构（SOES）。ReSSFormer以递归推理取代传统深度堆叠，以令牌与专家级稀疏注意力替代全连接注意力，并直接从内容建模潜在令牌拓扑。在语言建模、多跳问答及结构敏感任务中，ReSSFormer在相近的浮点运算量（FLOPs）与参数量预算下，持续优于强基线，展现出良好的可扩展性、高效性与结构灵活性。
原文摘要 · Abstract (English)
While Transformer architectures have demonstrated impressive scalability across domains, they continue to face challenges in long-context reasoning, computational efficiency, and structural generalization - largely due to rigid layer stacking, dense attention, and reliance on positional encodings. We present ReSSFormer, a Recursive Sparse Structured Transformer that integrates three complementary innovations: Recurrent Reasoning & Memory Unit (R2MU) for iterative reasoning with bounded depth, Adaptive Sparse Attention Module (ASAM) for efficient and focused context selection, and Self-Organizing Encoder Structure (SOES) for position-free structure induction. ReSSFormer replaces conventional depth stacking with recurrent inference, substitutes full attention with token- and expert-level sparsity, and models latent token topology directly from content. Across language modeling, multi-hop QA, and structure-sensitive tasks, ReSSFormer consistently outperforms strong baselines under comparable FLOPs and parameter budgets, highlighting its scalability, efficiency, and structural flexibility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。