让Transformer每层独立循环，更省参数且效果更好
Intra-Layer Recurrence in Transformers for Language Modeling
- 每层单独循环，而非整块重复处理
- 早期层多循环能取得最佳效果
- 适合追求轻量化模型的研究者
Transformer模型在自然语言处理中已达到新基准，但其深度增加导致参数量急剧上升。现有循环式Transformer通过多次重处理层来缓解此问题，但通常对整个层块无差别地应用循环。本文研究了层内循环（ILR），一种更精准的方法：在单次前向传播中选择性地对个别层施加循环。实验表明，将更多迭代分配给早期层可获得最优结果。这些发现表明，ILR为优化Transformer架构中的循环结构提供了有前景的方向。
原文摘要 · Abstract (English)
Transformer models have established new benchmarks in natural language processing; however, their increasing depth results in substantial growth in parameter counts. While existing recurrent transformer methods address this issue by reprocessing layers multiple times, they often apply recurrence indiscriminately across entire blocks of layers. In this work, we investigate Intra-Layer Recurrence (ILR), a more targeted approach that applies recurrence selectively to individual layers within a single forward pass. Our experiments show that allocating more iterations to earlier layers yields optimal results. These findings suggest that ILR offers a promising direction for optimizing recurrent structures in transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。