螺旋递归架构让Transformer分层学习依赖关系，提升效率。
SpiralFormer: Looped Transformers Can Learn Hierarchical Dependencies via Multi-Resolution Recursion
- 通过多分辨率递归，在不同尺度上迭代优化表示
- 160M到1.4B参数规模下，比传统方法更省参数和算力
- 适合追求高效推理的序列建模任务
递归（环形）Transformer通过重复应用共享层，将计算深度与参数深度解耦，为迭代精炼和隐式推理提供显式架构支持。然而，早期环形Transformer常在同等计算量下表现不及非递归基线。尽管近期研究引入了更有效的递归机制以缩小差距，现有架构仍固定于全令牌分辨率，忽视了在压缩隐状态上计算的潜在效率。本文提出SpiralFormer，一种采用多分辨率递归策略的环形Transformer。我们通过探测证据表明，多分辨率递归能促使模型在不同尺度上实现迭代功能专业化，从而学习层次化依赖。实验表明，SpiralFormer在160M至1.4B参数规模下，均优于环形与非环形基线，确立了序列分辨率作为递归架构扩展的新维度。
原文摘要 · Abstract (English)
Recursive (looped) Transformers decouple computational depth from parameter depth by repeatedly applying shared layers, providing an explicit architectural primitive for iterative refinement and latent reasoning. However, early looped Transformers often underperform non-recursive baselines of equal compute. While recent literature has introduced more effective recursion mechanisms to mitigate this gap, existing architectures still operate at a fixed, full-token resolution, neglecting the potential efficiency of computing over compressed latent representations. In this paper, we propose SpiralFormer, a looped Transformer that executes recurrence under a multi-resolution recursion schedule. We provide probing evidence that multi-resolution recursion enables the model to learn hierarchical dependencies by inducing iteration-wise functional specialization across different scales. Empirically, SpiralFormer achieves better parameter and compute efficiency than both looped and non-looped baselines across model scales from 160M to 1.4B, establishing sequence resolution as a potential axis for scaling recursive architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。