让大模型层数扩展可学习，训练更快更省资源。
LESA: Learnable LLM Layer Scaling-Up
- 用SVD发现层间参数规律，通过神经网络预测新增层参数
- 持续预训练时性能更优，计算成本低于一半
- 适合作为大模型扩展的通用方案，尤其适合资源有限场景
从头训练大型语言模型需要巨大算力，成本高昂。模型规模扩展提供了一种可行方案，即利用小模型参数构建大模型。然而，现有深度扩展方法依赖经验性规则复制层数，导致初始化质量差、持续预训练收敛慢。我们提出新型可学习深度扩展方法LESA：通过拼接各层参数并应用奇异值分解，挖掘层间潜在规律，证明层间参数可被学习。LESA使用神经网络预测相邻层间的插入参数，实现更优初始化与更快训练。实验表明，LESA优于现有基线，在持续预训练中性能更佳，且计算成本不足一半。大量分析验证其在不同模型规模和任务上的有效性。
原文摘要 · Abstract (English)
Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive. Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones. However, existing depth scaling-up methods rely on empirical heuristic rules for layer duplication, which result in poorer initialization and slower convergence during continual pre-training. We propose \textbf{LESA}, a novel learnable method for depth scaling-up. By concatenating parameters from each layer and applying Singular Value Decomposition, we uncover latent patterns between layers, suggesting that inter-layer parameters can be learned. LESA uses a neural network to predict the parameters inserted between adjacent layers, enabling better initialization and faster training. Experiments show that LESA outperforms existing baselines, achieving superior performance with less than half the computational cost during continual pre-training. Extensive analyses demonstrate its effectiveness across different model sizes and tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。