通过分层缩放优化大模型预训练，提升性能且不降低效率
Crown, Frame, Reverse: Layer-Wise Scaling Variants for LLM Pre-Training
- 设计三种新分层缩放方法，动态调整不同层的网络宽度和注意力头数
- 在180M参数预算下，所有模型收敛损失相近，性能优于均等配置基线
- 适合关注模型结构优化与高效预训练的研究者和工程师
基于Transformer的语言模型传统上采用统一（各向同性）的层大小，却忽视了不同深度层的功能差异及其计算需求。受分层缩放（LWS）和剪枝研究启发，我们提出三种新的LWS变体——Framed、Reverse和Crown，通过两或三点线性插值，在预训练阶段重新分配前馈网络宽度和注意力头数。我们在固定180M参数预算、50亿词训练数据上首次系统评估了LWS及其变体。所有模型达到相近的损失值，且性能优于同等成本的均等配置基线，同时未显著降低训练吞吐量。本工作是预训练中分层架构设计的初步探索，未来需在更大规模参数与数据上验证其潜力。
原文摘要 · Abstract (English)
Transformer-based language models traditionally use uniform (isotropic) layer sizes, yet they ignore the diverse functional roles that different depths can play and their computational capacity needs. Building on Layer-Wise Scaling (LWS) and pruning literature, we introduce three new LWS variants - Framed, Reverse, and Crown - that redistribute FFN widths and attention heads via two or three-point linear interpolation in the pre-training stage. We present the first systematic ablation of LWS and its variants, on a fixed budget of 180M parameters, trained on 5B tokens. All models converge to similar losses and achieve better performance compared to an equal-cost isotropic baseline, without a substantial decrease in training throughput. This work represents an initial step into the design space of layer-wise architectures for pre-training, but future work should scale experiments to orders of magnitude more tokens and parameters to fully assess their potential.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。