通过循环深度折叠让同一模块同时承担浅层和深层角色，提升训练效率。
Mobius Learning: Cyclic Depth Folding in Transformers
- 采用循环移位的块顺序，使相同模块在不同数据流中扮演深浅不同角色
- 在4个工人下，2.5B tokens训练中验证损失低于传统架构
- 适合内存受限的分布式训练，每个节点只需存储部分模块
基于Transformer的语言模型将计算沿有序深度轴组织，浅层与深层模块常发展出不同表征功能。我们挑战了角色必须固定于位置的传统观点，提出基于循环深度折叠的训练架构——Mobius Learning。不同数据流遵循循环偏移的块顺序，使得同一模块组对某些数据流在序列早期应用，对另一些在后期应用，从而在浅层和深层角色上均被优化，此现象称为深度角色叠加。令人意外的是，在4个工人、124M参数的GPT-2小模型、2.5B FineWeb token、Muon训练设置下，Mobius Learning在更多块序列遍历次数时达到更低的验证损失。这一反直觉结果表明，模块无需固定于单一浅层或深层角色，为基于循环深度折叠的新设计空间打开可能。关键优势在于其适配内存受限的分布式训练：原始数据保持本地，每个工作节点仅需存储一个模块组而非完整堆栈。
原文摘要 · Abstract (English)
Transformer-based language models organize computation along an ordered depth axis, where shallow and deep blocks often develop distinct representational roles. We challenge the conventional view that these roles must remain tied to a block's position in the ordered sequence. We introduce Mobius Learning, a training architecture based on cyclic depth folding, in which different data streams follow cyclically shifted block orders. The same block group is therefore applied early in the block sequence for some data streams and late for others, so it is optimized in both shallow and deep roles, a phenomenon we call depth-role superposition. Surprisingly, in four-worker experiments with a modded GPT-2 small (124M) model trained on 2.5B FineWeb tokens using Muon, Mobius Learning achieves lower validation loss than a fixed-order looped Transformer at larger numbers of Transformer block-sequence passes. This counterintuitive result shows that a block group need not remain confined to one fixed shallow or deep role within the block sequence and opens a new design space based on cyclic depth folding. Crucially, this structure makes Mobius Learning particularly well suited to memory-constrained distributed training: raw training data remain local, while each worker stores one block group rather than the complete Transformer block stack.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。