arXiv:2604.21106cs.LGcs.CL2026-04被引 12

量化循环Transformer中一次重复相当于多少独立参数,揭示其真实效率

How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models

  • 通过等深度实验测量循环次数的等效参数价值,发现循环增益非线性
  • 循环指数φ=0.46,表明4次循环仅等效于1.7倍独立块容量
  • 该指标可区分真实性能提升与训练缺陷,适合模型优化研究者使用

我们测量了在循环(深度递归)Transformer中,一次循环相当于多少独立参数。通过对循环次数r ∈ {1, 2, 4, 8}的等深度预训练实验,覆盖约50倍训练算力变化,拟合联合缩放律:L = E + A(N_once + r^φ N_rec)^{-α} + B D^{-β},测得循环等价指数φ = 0.46。直观上,φ表示将一个模块循环r次,在验证损失上等效于非循环模型中添加r个独立模块(φ=1)或无容量增益的重复运行(φ=0)。我们的φ=0.46介于两者之间,说明在相同训练算力下,用共享循环替换独立模块会导致验证损失上升。例如,当r=4时,410M的循环模型性能相当于580M的非循环模型,但训练成本等同于1B的非循环模型。我们通过两个案例验证φ作为诊断工具的有效性:常用截断反向传播使φ降至0.38,表明循环机制训练不足,尽管验证损失下降;而超连接可使φ升至0.65,体现真实容量提升。该方法能区分真正的循环优势与训练侧收益,这是原始验证损失无法做到的。

原文摘要 · Abstract (English)

We measure how much one recurrence is worth to a looped (depth-recurrent) transformer, in equivalent unique parameters. From an iso-depth pretraining sweep across recurrence counts $r \in \{1, 2, 4, 8\}$ spanning ${\sim}50\times$ in training compute, we fit a joint scaling law $L = E + A\,(N_\text{once} + r^φ N_\text{rec})^{-α} + B\,D^{-β}$ and measure a recurrence-equivalence exponent $φ= 0.46$. Intuitively, $φ$ tells us whether looping a block $r$ times is equivalent in validation loss to $r$ unique blocks of a non-looped model (full equivalence, $φ{=}1$) or to a single block run repeatedly with no capacity gain ($φ{=}0$). Our $φ= 0.46$ sits in between, so replacing unique blocks with shared recurrences increases validation loss at matched training compute. For example, at $r{=}4$ a 410M looped model performs on par with a 580M non-looped model, but incurs the training cost of a 1B non-looped one. We demonstrate the utility of $φ$ as a diagnostic tool on two case studies: commonly used truncated backpropagation lowers $φ$ to $0.38$, indicating that the loop mechanism is poorly trained under truncation, even though validation loss decreases. Conversely, hyperconnections raise $φ$ to $0.65$, a genuine capacity gain. Our method separates true loop improvements from training-side gains, a distinction raw validation loss cannot make.

循环模型缩放定律效率评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。