Looped Transformer 更强因架构带来更优损失曲面，利于复杂模式学习。
What Makes Looped Transformers Perform Better Than Non-Recursive Ones
- 通过损失曲面几何解释递归结构优势，提出河谷类型区分
- 发现循环架构促进陡峭河谷形成，提升收敛与泛化能力
- 提出分阶段训练策略，加速训练且性能相当
尽管循环Transformer（Looped-Attn）在复杂推理任务中通常优于标准Transformer（Single-Attn），但其机制仍不明确。本文从损失曲面几何角度出发，基于样本与赫森矩阵层面的动态观察，扩展河谷地形模型，区分平缓的U型谷与陡峭的V型谷。我们推测，循环架构诱导出河谷型的归纳偏置，使优化过程可通过谷间跳跃实现更好收敛，并更利于学习复杂模式。相较之下，非递归架构产生平坦河谷（River-U-Valley）。基于此，我们提出SHIFT（Staged HIerarchical Framework for Progressive Training），一种系统性训练策略，可加速循环Transformer训练并保持相近性能。
原文摘要 · Abstract (English)
While looped transformers (termed as Looped-Attn) often outperform standard transformers (termed as Single-Attn) on complex reasoning tasks, the mechanism for this advantage remains underexplored. In this paper, we explain this phenomenon through the lens of loss landscape geometry, inspired by empirical observations of their distinct dynamics at both sample and Hessian levels. To formalize this, we extend the River-Valley landscape model by distinguishing between U-shaped valleys (flat) and V-shaped valleys (steep). Based on empirical observations, we conjecture that the recursive architecture of Looped-Attn induces a landscape-level inductive bias towards River-V-Valley. This inductive bias suggest a better loss convergence along the river due to valley hopping, and further encourage learning about complex patterns compared to the River-U-Valley induced by Single-Attn. Building on this insight, we propose SHIFT (Staged HIerarchical Framework for Progressive Training), a principled training strategy that accelerates the training process of Looped-Attn while achieving comparable performances.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。