通过调控初始化率和权重衰减，提升大模型推理能力。
Scalable Complexity Control Facilitates Reasoning Ability of LLMs
- 用初始化率和权重衰减控制模型复杂度。
- 2.4B模型在1万亿词上预训练，性能显著提升。
- 适合关注模型可扩展性与推理优化的研究者。
近年来，大语言模型（LLMs）的推理能力迅速发展，促使人们关注更基础且可靠的方法以提升其泛化能力。本文表明，通过调整初始化率和权重衰减系数来控制模型复杂度，能一致地改善不同模型规模和数据规模下的缩放规律。实验对比了在1万亿词上预训练的2.4B模型在不同复杂度超参数下的基准性能。结果发现，固定初始化标准差不如保持恒定的初始化率（即标准差的指数）有效，后者使模型和数据规模的缩放规律下降更快。这些结果表明，复杂度控制是推动LLM持续发展的有前景方向。
原文摘要 · Abstract (English)
The reasoning ability of large language models (LLMs) has been rapidly advancing in recent years, attracting interest in more fundamental approaches that can reliably enhance their generalizability. This work demonstrates that model complexity control, conveniently implementable by adjusting the initialization rate and weight decay coefficient, improves the scaling law of LLMs consistently over varying model sizes and data sizes. This gain is further illustrated by comparing the benchmark performance of 2.4B models pretrained on 1T tokens with different complexity hyperparameters. Instead of fixing the initialization std, we found that a constant initialization rate (the exponent of std) enables the scaling law to descend faster in both model and data sizes. These results indicate that complexity control is a promising direction for the continual advancement of LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。