揭示大模型凭上下文学习的数学规律，解释为何规模越大越能零样本适应新任务。
Scaling Laws and In-Context Learning: A Unified Theoretical Framework
- 用统一理论连接模型规模与上下文学习能力，发现性能随深度、宽度等呈幂律增长。
- 证明在特定条件下，模型前向传播等价于梯度元学习，有效学习率与根号规模成反比。
- 预测关键规模阈值和最优结构比例，适合研究大模型原理与架构设计者阅读。
上下文学习（ICL）使大语言模型能在不更新参数的情况下，仅通过示例就适应新任务。尽管已有大量实证研究，但对规模化下ICL涌现的机理仍缺乏系统理解。本文提出一个统一的理论框架，将缩放定律与Transformer中ICL的涌现联系起来。分析表明，ICL性能随模型深度 $L$、宽度 $d$、上下文长度 $k$ 及训练数据量 $D$ 呈幂律变化，指数由任务结构决定。我们证明，在特定条件下，Transformer在前向传播中实现了基于梯度的元学习，有效学习率 $η_{ ext{eff}} = Θ(1/\\/sqrt{Ld})$。实验验证了临界规模下的突变现象，并推导出在固定参数预算 $N = Ld$ 下最优的深度-宽度分配：$L^* \propto N^{2/3}$，$d^* \propto N^{1/3}$。系统性实验在合成任务上验证了理论预测，测量的缩放指数与理论高度一致。该工作给出了ICL涌现的充分必要条件，并确立了Transformer进行上下文学习的根本计算极限。
原文摘要 · Abstract (English)
In-context learning (ICL) enables large language models to adapt to new tasks from demonstrations without parameter updates. Despite extensive empirical studies, a principled understanding of ICL emergence at scale remains more elusive. We present a unified theoretical framework connecting scaling laws to ICL emergence in transformers. Our analysis establishes that ICL performance follows power-law relationships with model depth $L$, width $d$, context length $k$, and training data $D$, with exponents determined by task structure. We show that under specific conditions, transformers implement gradient-based metalearning in their forward pass, with an effective learning rate $η_{\text{eff}} = Θ(1/\sqrt{Ld})$. We demonstrate sharp phase transitions at critical scales and derive optimal depth-width allocations favoring $L^* \propto N^{2/3}$, $d^* \propto N^{1/3}$ for the fixed parameter budget $N = Ld$. Systematic experiments on synthetic tasks validate our predictions, with measured scaling exponents closely matching theory. This work provides both necessary and sufficient conditions for the emergence of ICLs and establishes fundamental computational limits on what transformers can learn in-context.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。