arXiv:2606.17945cs.AI2026-06

小初始化能显著提升大模型训练效果,尤其增强推理能力。

Small Initialization Matters for Large Language Models

论文配图:Small Initialization Matters for Large Language Models
图 1 · 摘自论文原文
  • 采用小参数初始化,引导模型经历压缩-扩展的发育路径
  • 在需要推理的任务上,小初始化带来最大性能提升
  • 建议默认使用小初始化,几乎无成本却有效

大语言模型为理解智能如何涌现提供了可操作的系统,而不仅限于工程优化。尽管通常归因于规模、数据和架构的进步,我们发现参数初始化是决定训练效果与模型容量的基因级因素。减小初始化尺度能持续提升预训练表现,尤其在高推理需求任务中收益显著。我们识别出两个普遍存在的经验设置限制了小初始化的优势,并证明放宽它们可恢复有利的缩放规律。进一步揭示了一种关键初始化,能在推理与训练间取得平衡。机制上,小初始化驱动参数先凝聚成低复杂度结构,再扩展为丰富表征,具体体现了‘压缩即智能’的理念。逐标记分析显示,性能提升集中于非平凡、上下文受限的预测,而非均匀分布于所有标记。这些结果启发了一个简单的γ-初始化规则:将初始化范围作为显式控制旋钮,始终采用小初始化,默认策略几乎零成本,却能提升预训练并增强跨规模的推理能力。

原文摘要 · Abstract (English)

Large language models provide a tractable system for asking how intelligence itself emerges, rather than only how LLMs can be engineered. Although progress is usually attributed to scale, data and architecture, we show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity. Reducing the initialization scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks. We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling. We further uncover a critical initialization that balances the reasoning and training. Mechanistically, small initialization drives a distinct developmental trajectory: parameters first condense into low-complexity structures and later expand into richer representations, giving concrete form to the idea that compression is intelligence. Token-level analyses show that the gains concentrate on non-trivial, context-constrained predictions rather than all tokens uniformly. These results motivate a simple $γ$-initialization rule: expose initialization rage as an explicit knob and use small initialization by default, an almost cost-free intervention that improves pretraining and strengthens reasoning across model scales.

大模型初始化推理能力训练优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。