提出分析循环Transformer稳定性的三轴框架,揭示记忆与泛化的关键机制。
Stability and Generalization in Looped Transformers

- 基于不动点构建三维度稳定性分析框架:可达性、输入依赖性、几何特性。
- 无记忆的循环网络仅有可数不动点,无法实现强输入依赖;引入外归一化与记忆后可实现稳定预测。
- 实验证明性能与框架预测一致,新提出的内部记忆放置在数独任务中表现更优。
循环Transformer通过测试时迭代提升计算能力,但其能否泛化到更难问题仍不明确。本文提出基于不动点的分析框架,从可达性、输入依赖性和几何特性三个维度刻画循环结构的稳定性,并证明:无记忆的循环网络仅有可数个不动点,无法在任意谱域实现强输入依赖;而结合外归一化与记忆机制可实现不动点的可到达性、输入局部平滑性及稳定反向传播。我们在国际象棋、数独和前缀求和任务上训练单层循环Transformer,发现下游性能与框架预测高度一致。此外,我们提出一种新的内部记忆放置方式,在应用外归一化后,其表现不仅与标准放置相当,且在数独任务中显著更优。
原文摘要 · Abstract (English)
Looped transformers promise test-time compute scaling by spending more iterations on harder problems, but it remains unclear which architectural choices let them extrapolate to harder problems at test time rather than memorize training-specific solutions. We introduce a fixed-point based framework for analyzing looped architectures along three axes of stability -- reachability, input-dependence, and geometry -- and use it to characterize when fixed-point iteration yields meaningful predictions. Theoretically, we prove that looped networks without recall have countable fixed points and cannot achieve strong input-dependence at any spectral regime, while recall combined with outer normalization reliably produces a regime in which fixed points are simultaneously reachable, locally smooth in the input, and supported by stable backpropagation. Empirically, we train single-layer looped transformers on chess, sudoku, and prefix-sums and find that downstream performance tracks the framework's predictions across tasks and architectural configurations. We additionally introduce internal recall, a novel recall placement variant, and show that it becomes competitive with -- and on sudoku, substantially better than -- standard recall placement once outer normalization is applied.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。