解析线性Transformer的上下文学习动态，揭示其分阶段学习机制。
Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application to Non-Linear Transformers
- 通过闭式SGD推导线性Transformer的精确学习动力学。
- 发现输入数据协方差决定时间尺度，出现阶段性学习与固定点行为。
- 提出谱秩动态等新指标，解释注意力网络中ICL突现与延迟泛化现象。
Transformer模型展现出强大的上下文学习（ICL）能力，能从上下文中的示例中适应新任务，但其内在机制仍不清晰。本文通过推导简化线性Transformer在回归任务上的闭式随机梯度下降（SGD）动力学，首次实现对ICL涌现的精确解析。分析揭示：（1）自然的时间尺度分离由输入数据协方差结构直接决定，导致分阶段学习；（2）精确描述了ICL的发展过程，包括对应于已学习算法的固定点及约束动力学的守恒律；（3）尽管模型为线性，却表现出意外的非线性学习行为。我们推测该现象可推广至非线性模型。为此，引入受理论启发的宏观度量（如谱秩动态、子空间稳定性），用于解释（1）仅含注意力的网络中ICL的突然涌现，以及（2）模算术模型中的延迟泛化（grokking）。本工作提供了ICL的精确动力学模型，并为分析复杂Transformer训练提供理论工具。
原文摘要 · Abstract (English)
Transformer models exhibit remarkable in-context learning (ICL), adapting to novel tasks from examples within their context, yet the underlying mechanisms remain largely mysterious. Here, we provide an exact analytical characterization of ICL emergence by deriving the closed-form stochastic gradient descent (SGD) dynamics for a simplified linear transformer performing regression tasks. Our analysis reveals key properties: (1) a natural separation of timescales directly governed by the input data's covariance structure, leading to staged learning; (2) an exact description of how ICL develops, including fixed points corresponding to learned algorithms and conservation laws constraining the dynamics; and (3) surprisingly nonlinear learning behavior despite the model's linearity. We hypothesize this phenomenology extends to non-linear models. To test this, we introduce theory-inspired macroscopic measures (spectral rank dynamics, subspace stability) and use them to provide mechanistic explanations for (1) the sudden emergence of ICL in attention-only networks and (2) delayed generalization (grokking) in modular arithmetic models. Our work offers an exact dynamical model for ICL and theoretically grounded tools for analyzing complex transformer training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。