比较三种新型序列模型，xLSTM在复杂任务中表现最优。
On Subquadratic Architectures: From Applications to Principles
- 对比xLSTM、Mamba-2和Gated DeltaNet三种子二次架构
- xLSTM在代码和时序预训练任务中综合表现最佳
- 其优势源于更灵活稳定的记忆修正机制
Transformer在现代序列建模中占主导地位，但其二次注意力带来高昂计算成本。子二次架构提供了可扩展的替代方案，但何种设计更有效仍不明确。本文比较了三种领先方法：xLSTM、Mamba-2 和 Gated DeltaNet。在具有复杂依赖关系的任务上进行评估：(1) 代码模型预训练，(2) 从大语言模型蒸馏代码模型，(3) 时间序列基础模型预训练。在这些设置中，xLSTM展现出最强的整体性能。为解释其优势，我们提出统一公式并分析底层架构机制，重点关注状态追踪与记忆动态。结果表明，xLSTM通过其门控机制实现更灵活且稳定的记忆修正。我们在受控的合成长度泛化任务上验证了这些发现。总体而言，xLSTM在复杂任务中的提升源自稳健的状态追踪与累积能力。
原文摘要 · Abstract (English)
Transformers dominate modern sequence modeling, but their quadratic attention incurs substantial computational cost. Subquadratic architectures offer a scalable alternative. However, it remains unclear which designs yield the most effective sequence models. We compare three leading approaches: xLSTM, Mamba-2, and Gated DeltaNet. We evaluate these models on tasks with complex dependencies: (1) code-model pre-training, (2) distillation of code models from large language models, and (3) pre-training of time-series foundation models. Across these settings, xLSTM delivers the strongest overall performance. To explain xLSTM's advantage, we present a unified formulation and analyze the underlying architectural mechanisms, focusing on state tracking and memory dynamics. Our results show that xLSTM enables more flexible and stable memory correction via its gating scheme. We corroborate these findings on controlled synthetic length-generalization tasks. Overall, our findings indicate that xLSTM's gains on complex tasks stem from robust state tracking and accumulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。