通过注入新层高效扩展大模型，无需大量训练即可保持性能。
Transformer Layer Injection: A Novel Approach for Efficient Upscaling of Large Language Models
- 在每K层间插入新Transformer层，减少训练干扰
- 1B~8B模型上初始损失更低，训练步数减少30%以上
- 适合需要低成本扩展模型的工程团队
本文提出Transformer Layer Injection（TLI），一种高效扩展大语言模型（LLM）的新方法，可在降低计算成本的同时维持模型性能。模型规模是提升机器学习质量的关键因素，而TLI通过降低初始损失、减少微调需求并保持模型复杂度来应对扩展挑战。该方法在每K个层组中注入新层，使隐藏表示能以最小扰动通过Transformer模块。我们在小规模模型（LLama3 1B、3B、8B）上对比了TLI与Mixture of Experts（MoE）和传统深度扩展（DUS）方法，结果表明：TLI实现更优初始化，训练步骤更少，且在KoBEST和KMCQA任务上表现更佳，部分模型甚至无需额外训练即可有效运行。实验验证了TLI在数据效率与成本效益上的显著优势，其可扩展性与简洁性使其成为从10B到405B参数模型扩展的有前景方案。
原文摘要 · Abstract (English)
In this paper, we propose Transformer Layer Injection (TLI), a novel method for efficiently upscaling large language models (LLMs) while minimizing computational costs and maintaining model performance. Model scale is a key factor in enhancing the quality of machine learning models, and TLI addresses the challenge of scaling by reducing initial loss, minimizing fine-tuning requirements, and preserving model complexity. Our approach improves upon the conventional Depth Up-Scaling (DUS) technique by injecting new layers into every set of K layers, enabling hidden representations to pass through transformer blocks with minimal disruption. We compare TLI with existing approaches, including Mixture of Experts (MoE) and DUS, and validate its efficiency through experiments on small LLMs (LLama3 1B, 3B, and 8B). Results show that TLI achieves better initialization, requires fewer training steps, and delivers superior accuracy on tasks such as KoBEST and KMCQA, with models performing effectively even without additional training. TLI is demonstrated to be both data-efficient and cost-effective, significantly outperforming existing methods. Its scalability and simplicity make it a promising solution for upscaling transformer-based models, with potential applications in scaling models from 10B to 405B parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。