无需注意力机制的模型在复杂推理上超越大模型。
Scaling Reasoning without Attention
- 用状态空间层替代注意力，实现固定内存、常数时间推理。
- 70亿参数模型在AIME和Livecodebench上超越更大模型。
- 适合追求高效推理与可扩展性的高难度推理任务研究者。
大型语言模型在复杂推理任务中取得显著进展，但仍受限于架构效率低下（依赖Transformer）和高难度领域缺乏结构化微调。我们提出 heirmodel，一种无注意力机制的语言模型，通过架构与数据驱动创新解决上述问题。基于Mamba-2的状态空间双层（SSD），该模型摒弃自注意力与键值缓存，实现固定内存、常数时间推理。为训练其处理复杂推理，我们提出基于 extsc{PromptCoT}合成范式的两阶段课程微调策略，通过抽象概念选择与推理引导生成，构建教学结构化问题。基准测试显示， heirmodel-7B在同等规模下优于强健的Transformer及混合模型，并在AIME 24上超过更庞大的Gemma3-27B 2.6%，在AIME 25上领先0.6%，在Livecodebench上领先3.0%。结果表明，状态空间模型是注意力架构在高容量推理中的高效可扩展替代方案。
原文摘要 · Abstract (English)
Large language models (LLMs) have made significant advances in complex reasoning tasks, yet they remain bottlenecked by two core challenges: architectural inefficiency due to reliance on Transformers, and a lack of structured fine-tuning for high-difficulty domains. We introduce \ourmodel, an attention-free language model that addresses both issues through architectural and data-centric innovations. Built on the state space dual (SSD) layers of Mamba-2, our model eliminates the need for self-attention and key-value caching, enabling fixed-memory, constant-time inference. To train it for complex reasoning, we propose a two-phase curriculum fine-tuning strategy based on the \textsc{PromptCoT} synthesis paradigm, which generates pedagogically structured problems via abstract concept selection and rationale-guided generation. On benchmark evaluations, \ourmodel-7B outperforms strong Transformer and hybrid models of comparable scale, and even surpasses the much larger Gemma3-27B by 2.6\% on AIME 24, 0.6\% on AIME 25, and 3.0\% on Livecodebench. These results highlight the potential of state space models as efficient and scalable alternatives to attention-based architectures for high-capacity reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。