用可学习的固定点迭代提升语言模型性能,训练更稳、推理更快。
Solve the Loop: Attractor Models for Language and Reasoning

- 先生成输出嵌入,再通过求解固定点迭代优化,梯度由隐式微分获得。
- 770M模型胜过1.3B Transformer,推理任务准确率超90%,训练成本更低。
- 能将迭代过程内化为初始状态,推理时无需迭代,适合小模型部署。
循环Transformer通过迭代优化隐藏表示,提升语言建模与推理能力,但其训练不稳定、优化与部署成本高,且递归深度固定。本文提出吸引子模型(Attractor Models),先由主干模块生成输出嵌入,再由吸引子模块通过求解固定点进行优化,并利用隐式微分获取梯度。因此训练中有效深度恒定,迭代次数可自适应选择。实验表明,该模型在大规模语言预训练和小模型推理两个场景均优于现有方法:在语言建模中,困惑度最高降低46.6%,下游任务准确率提升19.7%,且训练成本更低;一个仅770M参数的模型表现超过训练了两倍数据的1.3B Transformer。在挑战性推理任务上,27M参数模型仅用约1000个样本即达到Sudoku-Extreme 91.4%与Maze-Hard 93.1%准确率,优于前沿模型如Claude和GPT-o3,也克服了传统递归推理器在规模扩大时的崩溃问题。此外,发现一种新现象——平衡内化:固定点训练使初始输出接近平衡状态,推理时可移除求解器而性能损失极小。这些结果表明,吸引子模型通过让模型学会内化迭代过程,实现了可扩展的迭代优化。
原文摘要 · Abstract (English)
Looped Transformers offer a promising alternative to purely feed-forward computation by iteratively refining latent representations, improving language modeling and reasoning. Yet recurrent architectures remain unstable to train, costly to optimize and deploy, and constrained to small, fixed recurrence depths. We introduce Attractor Models, in which a backbone module first proposes output embeddings, then an attractor module refines them by solving for the fixed point, with gradients obtained through implicit differentiation. Thus, training memory remains constant in effective depth, and iterations are chosen adaptively by convergence. Empirically, Attractor Models outperform existing models across two regimes, large-scale language-model pretraining and reasoning with tiny models. In language modeling, Attractor Models deliver a Pareto improvement over standard Transformers and stable looped models across sizes, improving perplexity by up to 46.6% and downstream accuracy by up to 19.7% while reducing training cost. Notably, a 770M Attractor Model outperforms a 1.3B Transformer trained on twice as many tokens. On challenging reasoning tasks, we show that our model with only 27M parameters and approximately 1000 examples achieves 91.4% accuracy on Sudoku-Extreme and 93.1% on Maze-Hard, scaling favorably where frontier models like Claude and GPT o3, fail completely, and specialized recursive reasoners collapse at larger sizes. Lastly, we show that Attractor Models exhibit a novel phenomenon, which we call equilibrium internalization: fixed-point training places the model's initial output embedding near equilibrium, allowing the solver to be removed at inference time with little degradation. Together, these results suggest that Attractor Models make iterative refinement scalable by turning recurrence into a computation the model can learn to internalize.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。