对比分层循环与堆叠层，发现前者在语言建模上表现明显更差。
Hierarchical vs. Flat Iteration in Shared-Weight Transformers
- 用快慢模块分步递归替代多层堆叠，参数共享但结构不同
- 1.2B参数模型在五次实验中均显示分层循环效果显著低于堆叠层
- 适合研究模型结构对表征能力影响的从业者参考
我们对基于Transformer的语言模型中，分层共享权重的递归结构是否能媲美独立层堆叠的表征能力进行了实证研究。HRM-LM将L个独立的Transformer层替换为双速递归结构：快速模块每步运行以进行局部优化，慢速模块每T步运行一次以实现全局压缩。该递归层次结构展开M = N × T步，参数共享。核心且稳健的发现是，经由五个独立运行的参数匹配型通用Transformer（UniTF, 1.2B）验证，两种方法间存在显著的性能差距。
原文摘要 · Abstract (English)
We present an empirical study of whether hierarchically structured, shared-weight recurrence can match the representational quality of independent-layer stacking in a Transformer-based language model. HRM-LM replaces L independent Transformer layers with a two-speed recurrent pair: a Fast module operating at every step for local refinement, and a Slow module operating every T steps for global compression. This recurrent hierarchy is unrolled for M = N x T steps with shared parameters. The central and most robust finding, supported by a parameter-matched Universal Transformer ablation (UniTF, 1.2B) across five independent runs, is a sharp empirical gap between the two approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。