揭示AdamW优化器中梯度扰动的延迟影响机制
Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW
- 将AdamW建模为有限时域输入-状态-输出系统，捕捉优化器记忆效应
- 推导出多步误差分解公式，在局部光滑条件下实现一阶精度
- 实验验证了响应机制与状态依赖影响，适合研究优化器动态的学者
小批量梯度更新会通过AdamW优化器的状态存储，对训练产生超出当前更新时刻的延迟影响。本文通过仅在一次梯度更新上差异的成对轨迹，研究这一延迟效应。将AdamW建模为包含模型参数及一阶、二阶矩估计的有限时域输入-状态-输出（ISO）系统。线性化联合动力学后，得到一个带符号的响应算子，可将局部梯度扰动映射为未来的损失变化，揭示优化器记忆如何塑造其幅度、时间与符号。进一步推导出精确的多步误差分解，并在局部光滑和受控激活切换条件下建立一阶有限时域精度。实验验证了响应机制与优化器状态的影响，重复未来分析揭示了显著的前瞻性结构，该结构部分可通过ISO近似恢复。代码已开源：https://github.com/Kanyooo/Loss_ISO。
原文摘要 · Abstract (English)
A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence. We formulate AdamW as a finite-horizon input--state--output (ISO) system whose state contains the model parameters and first- and second-moment estimates. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign. We further derive an exact multistep error decomposition and establish first-order finite-horizon accuracy under local smoothness and controlled activation switching. Experiments validate the response mechanism and optimizer-state effects, while repeated-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations. Code is available at https://github.com/Kanyooo/Loss_ISO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。