循环语言模型的监督机制存在盲区,仅靠密集监督无法控制隐藏状态的尺度。
Dense Supervision Is Not Enough: The Readout Blind Spot in Looped Language Models

- 通过读出层暴露的状态变量受损失函数控制,而非所有循环中活跃变量
- 使用RMSNorm等尺度不变读出时,隐藏状态范数仍可飙升至数千甚至数万
- 需让尺度对损失可见或从循环中移除,才能有效控制状态规模
循环语言模型将隐藏状态转化为运行时状态:每个状态被解码预测并反馈至后续计算。这引发基本监督问题:交叉熵损失实际控制哪些状态变量?我们发现,每轮密集交叉熵仅控制读出层暴露的变量,而非循环转移中所有活跃变量。隐藏状态的尺度成为具体失效模式:如RMSNorm和LayerNorm等尺度不变读出会隐藏径向尺度,而预归一化残差递归仍持续传递并更新该尺度。因此,即使无跨轮归一化,每轮损失仍可使44M和129M循环Transformer的最终隐藏状态范数达数千至数万。使用尺度可见读出或显式范数惩罚可将范数维持在十以内,而移除循环中的尺度则是互补性架构修复方案。由此得出设计原则:密集监督可训练退出点;而循环尺度控制需使尺度对损失可见或从循环中移除。符合此原则的变体在可变深度基准测试中,于相同推理深度下达到更低困惑度。
原文摘要 · Abstract (English)
Looped language models turn hidden states into runtime state: each state is decoded for prediction and fed back into future computation. This creates a basic supervision question: which state variables does cross-entropy actually control? We show that dense per-loop cross-entropy controls the variables exposed by the readout, not every variable active in the recurrent transition. Hidden-state scale gives a concrete failure mode. Scale-invariant readouts such as RMSNorm and LayerNorm hide radial scale from the immediate cross-entropy loss, while pre-norm residual recurrence continues to carry and update that same scale. Thus per-loop loss can make early exits usable without controlling recurrent scale. In 44M and 129M looped transformers without inter-loop normalization, per-loop cross-entropy through RMSNorm readouts still drives final hidden-state norms into the thousands or tens of thousands. Scale-visible readouts and explicit norm penalties keep norms in the tens, and scale-removing recurrence is the complementary architectural fix. The resulting design rule is simple: dense supervision trains exits; recurrent scale control requires either making scale visible to a loss or removing it from the loop. Consistent with this rule, scale-controlled variants achieve lower perplexity at matched inference-depth operating points in our variable-depth benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。