arXiv:2607.27656cs.LGcs.CL2026-07

提出SCSE机制,让循环Transformer在保持输入特征的同时稳定递归状态。

Looped Transformers with Source-Centered State Evolution

论文配图:Looped Transformers with Source-Centered State Evolution
图 1 · 摘自论文原文
  • 设计锚点与偏差递归,使输入依赖与状态不变性共存
  • 在多个数据集上提升循环深度下的生成质量,尤其在长文本任务中优势明显
  • 适合研究高效递归架构或模型稳定性问题的从业者

循环Transformer通过重复使用同一Transformer模块,在固定参数量下增加有效深度,实现训练与推理时的计算弹性。然而,共享模块需处理不断变化的隐藏状态轨迹,且在加性注入型循环结构中,每步重引入输入相关信号,可能导致状态偏移。本文提出源中心状态演化(SCSE),通过学习锚点与初始偏差,保留输入依赖性;允许非零偏差驱动递归计算,同时确保零偏差映射为零,并以零偏差掩码保障锚点精确不变性。该设计使锚点成为一步不动点。理论表明,零偏差强迫偏置是可调节的设计自由度,其影响可正可负;SCSE将其设为零,以实现严格锚点不变性。在WikiText-2、WikiText-103、直接网页语料预训练、未见网页文本迁移及LAMBADA填空任务中,SCSE均提升受控递归质量边界。消融实验表明,学习锚点与锚坐标偏差递归是性能提升主因,模型案例研究验证了锚响应诊断与实际递归运动的一致性。

原文摘要 · Abstract (English)

Looped Transformers create a useful train- and test-time compute axis by reusing the same Transformer block over recurrent depth, increasing effective depth at a fixed parameter count. However, that shared block must then govern an entire trajectory of varying hidden states over trained and extrapolated depths. Furthermore, in additive-injection looped Transformers, an input-conditioned signal is reintroduced at every recurrent step, so applying the shared transition at an input-conditioned reference can still move the hidden state. In this paper, we propose Source-Centered State Evolution (SCSE), which is designed to reconcile input conditioning with reference-preserving shared recurrence. Specifically, SCSE retains input dependence through its learned anchor and initial deviation, allows nonzero deviations to drive recurrent computation while mapping zero deviation to zero, and guarantees exact anchor invariance through its zero-deviation mask. The designated anchor is thereby a one-step fixed point by construction. The zero-deviation forcing bias is the next deviation produced from the anchor itself and vanishes in SCSE, while nonzero deviations remain active and support state-dependent recurrent computation. Our theory shows that the zero-deviation forcing bias is a design degree of freedom whose task effect can be harmful, neutral, or beneficial; SCSE resolves this choice in favor of exact anchor invariance by setting the bias to zero. Across WikiText-2, WikiText-103, direct web-corpus pretraining, held-out web-text transfer, and LAMBADA completion, SCSE improves the controlled recurrent quality frontier. Ablation studies identify the learned anchor and the anchor-coordinate deviation recurrence as the primary contributors to the gain, and a trained-model case study grounds the anchor-response diagnostic in observed recurrent motion.

循环神经网络Transformer状态演化模型稳定性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。