提出可自适应计算量的循环型Transformer模型，解决深度推理中的信号衰减问题。
Fixed-Point Reasoners: Stable and Adaptive Deep Looped Transformers
- 采用预归一化和残差缩放，提升循环架构的信号传播能力。
- 通过固定点收敛实现端到端停顿机制，计算量随任务难度自动调整。
- 在数独、迷宫等推理任务上表现优异，适合复杂逻辑推理场景。
循环架构为需要组合推理的任务提供了逐步求解的归纳偏置。循环次数决定模型所能达到的有效层数，进而影响解的质量。与深层网络类似，循环架构也因停顿决策被推迟而面临信号传播问题。本文通过引入预归一化层和残差缩放解决该问题。基于这些结构改进，提出FPRM——一种基于Transformer的固定点推理模型，利用固定点收敛作为循环架构中的端到端停顿机制。实验表明，固定点停顿使FPRM能根据任务难度自适应调整计算量。FPRM在常见推理基准测试中表现良好，包括Sudoku、Maze、状态追踪和ARC-AGI。
原文摘要 · Abstract (English)
Looped architectures provide an inductive bias toward learning step-by-step procedures for tasks that require compositional reasoning. The number of effective layers reached by looping determines the quality of the solution these models find. Like deep architectures, looped architectures are prone to a signal propagation problem induced by depth as the halting decision is postponed. In this paper, we address this signal propagation issue using pre-norm layers and residual scaling. Building on these architectural modifications, we propose FPRM, a Transformer-based Fixed-Point Reasoning Model that uses fixed-point convergence as an end-to-end halting mechanism in a looped architecture. We show that fixed-point halting allows FPRM to adapt its compute to task difficulty. FPRM is effective on common reasoning benchmarks, namely Sudoku, Maze, state-tracking, and ARC-AGI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。