提出双尺度动态机制，提升循环深度变压器的推理效率与稳定性。
Two-Scale Latent Dynamics for Recurrent-Depth Transformers
- 将迭代过程分为微观精修与宏观漂移两个尺度，揭示内在动态规律。
- 迭代步长逐渐变小且趋于正交，表明模型在精细结构上建模更优。
- 基于二阶步长差的早停策略优于已有方法，兼顾性能、稳定与速度。
循环深度变压器通过在生成令牌前迭代潜变量计算来扩展测试时计算量。我们研究了这些迭代点的几何特性，提出一个简单的双尺度操作视角：(i) 在循环块内，更新作为小尺度精修；(ii) 在连续块间，状态经历更大尺度漂移。训练过程中，我们的测量显示循环步长逐渐减小，且彼此间趋于正交，表明模型对细粒度结构的局部建模能力增强，而非仅沿单一方向推进。这一动态启发我们设计一种基于模型二阶步长差的早停机制，相比Geiping等人提出的KL散度退出策略及其朴素的一阶版本，在性能、稳定性和时间效率上均表现更优。
原文摘要 · Abstract (English)
Recurrent-depth transformers scale test-time compute by iterating latent computations before emitting tokens. We study the geometry of these iterates and argue for a simple, two-scale operational picture: (i) within a looped block, updates act as small-scale refinements; (ii) across consecutive blocks, states undergo a larger-scale drift. Across training, our measurements show that loop steps become smaller and increasingly orthogonal to one another, indicating better local modeling of fine structure rather than merely pushing in a single direction. These dynamics motivate an early-exit mechanism based on the model's second-order difference in step-size, which we show is superior in terms of performance, stability and time-efficiency, when compared to the KL-divergence exit strategy of Geiping et al. and its naive first-order counterpart.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。