提出可扩展的门控增量网络特征学习方法，实现跨模型宽度的稳定学习率迁移。
Unlocking Feature Learning in Gated Delta Networks at Scale
- 通过前向传播与门控机制推导出门控增量网络的缩放规律
- 在语言模型预训练中验证了学习率跨宽度迁移的稳定性
- 适用于大规模线性结构模型，适合追求高效训练的研究者
训练和扩展大语言模型需要巨大的计算资源，推动了高效亚二次架构和合理超参数调优方法的发展。尽管最大更新参数化（μP）已实现标准Transformer的零样本超参数迁移，但其在具有结构化状态转移和复杂架构的线性模型中的扩展仍基本未被探索。通过严格地在前向传播、门控机制和递归状态动态中传递坐标尺度估计，我们推导出门控增量网络的缩放规则。在语言模型预训练实验中证实，我们的配置在AdamW和SGD下均能实现模型宽度间的稳定学习率迁移，而标准参数化则无法实现，验证了分析的正确性和实际有效性。
原文摘要 · Abstract (English)
Training and scaling Large Language Models demand enormous computational resources, motivating both efficient sub-quadratic architectures and principled hyperparameter tuning methods. While the Maximal Update Parametrization ($μ$P) has enabled zero-shot hyperparameter transfer for standard Transformers, its extension to linear models, particularly those with structured state transitions and complicated architectures, remains largely unexplored. By rigorously propagating coordinate-size estimates through the forward pass, gating mechanisms, and recurrent state dynamics, we derive the scaling rules for Gated Delta Network. Experiments on language-model pre-training confirm that our configurations enable stable learning-rate transfer across model widths under both AdamW and SGD, whereas standard parametrization fails to transfer, validating the correctness and practical utility of our analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。