提出SPARKLING框架，实现模型宽度渐进扩展中的稳定性与多样性平衡。
SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
- 通过均方根尺度一致性保持激活统计稳定
- 采用非对称优化器重置和学习率热身打破梯度对称性
- 在多种模型架构上降低35%训练成本，适合大规模模型训练
渐进式学习（PL）通过逐步增加模型规模来减少预训练计算开销。尽管深度扩展已得到广泛研究，宽度扩展仍严重不足，现有方法仅限于训练早期。然而，在中期扩展宽度对最大化计算节省至关重要，却因严重训练不稳定性而难以实现。实验表明，该阶段直接初始化会破坏激活统计，引发损失飙升；而复制初始化则引入梯度对称性，抑制特征多样性。为此，我们提出SPARKLING（平衡信号保留与对称性打破的宽度渐进学习），通过均方根尺度一致性稳定激活统计，利用非对称优化器状态重置与非对称学习率热身确保对称性打破。在密集模型与混合专家（MoE）模型上的大量实验表明，跨多个宽度轴与优化器族，SPARKLING始终优于从头训练，并在宽度扩大2倍时将训练成本降低最多达35%。
原文摘要 · Abstract (English)
Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansion remains significantly understudied, with the few existing methods limited to the early stages of training. However, expanding width during the mid-stage is essential for maximizing computational savings, yet it remains a formidable challenge due to severe training instabilities. Empirically, we show that naive initialization at this stage disrupts activation statistics, triggering loss spikes, while copy-based initialization introduces gradient symmetry that hinders feature diversity. To address these issues, we propose SPARKLING (balancing {S}ignal {P}reservation {A}nd symmet{R}y brea{K}ing for width-progressive {L}earn{ING}), a novel framework for mid-stage width expansion. Our method achieves signal preservation via RMS-scale consistency, stabilizing activation statistics during expansion. Symmetry breaking is ensured through asymmetric optimizer state reset and asymmetric learning rate re-warmup. Extensive experiments on dense and Mixture-of-Experts (MoE) models demonstrate that, across multiple width axes and optimizer families, SPARKLING consistently outperforms training from scratch and reduces training cost by up to 35% under $2\times$ width expansion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。