用循环调制让少量层实现多层效果,提升模型效率与性能。
Gated Recurrent Transformers: Expressive Depth through Recurrent Modulation

- 设计循环调制结构,用共享核心重复计算,实现深度表达力。
- 在相同算力下,3层模型达到12层GPT-2的准确率,降低59%内存峰值。
- 适合追求高效率、低资源的大型语言模型部署场景。
扩大Transformer语言模型时,表达能力与内存效率存在内在矛盾。独立参数的多层结构虽保留功能分化但内存开销大;而常规深度共享导致表征单一,性能下降。本文提出门控循环Transformer(Gated Recurrent Transformer),由固定前缀和尾部模块夹住一个可迭代R次的共享核心。受门控循环网络启发,引入轻量投影与基于隐藏状态、前缀输出及每步重采样噪声的逐元素更新门,动态调节循环更新。该机制使少数共享层在多次迭代中实现功能分化,无需大量独立层。在等FLOPs约束下,3层模型性能媲美12层GPT-2 Small,九个规模-预算组合中均优于MoR与重尾深度采样;中大规模下接近密集模型质量,预算加倍后超越之。在等参数约束下,更深循环实现2.76验证损失,优于非循环对应模型的2.84。结果表明,自适应深度复用是用参数换质量的合理策略:大规模下参数减少63%,解码峰值内存降低59%,仅增加10%编译生成延迟。
原文摘要 · Abstract (English)
Scaling transformer language models creates an inherent tension between expressivity and memory efficiency. While unique weights across layers preserve functional specialization---from input-grounding to abstract refinement---they incur a substantial memory footprint. Conversely, standard depth-sharing enforces uniform transformations that collapse representational diversity and degrade modeling quality. We introduce Gated Recurrent Transformer, a recurrent depth transformer where fixed-depth prelude and coda blocks bracket a single shared core iterated R times. Inspired by gated recurrent neural networks, we employ a lightweight projection and an elementwise update gate---conditioned on the hidden state, the fixed prelude output, and noise resampled at every step---to modulate the recurrent update. This allows the model to specialize the input to the same few layers across recurrences, rather than requiring many unique layers to achieve functional diversity. Under an isoFLOPS constraint, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline with similar training and inference FLOPs, and leads MoR and heavy-tail depth sampling in all nine scale-by-budget cells; at medium and large scale it approaches dense quality at the standard token budget and overtakes it at medium scale once that budget is doubled. Under an isoPARAMS constraint, deeper recurrence achieves a 2.76 validation loss versus 2.84 for a non-recurrent counterpart at matched parameter and data budget. Our results demonstrate that adaptive depth reuse is a principled strategy for trading parameters for quality: at large scale, 63% fewer parameters and 59% less peak decoding memory for a 10% increase in compiled generation latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。