高阶优化器看似更优，实则代价高昂且无泛化优势。
Adaptive Runge-Kutta Step Control Buys Training Loss, Not Generalization: An Honest Compute-Matched Study of RK-Adam Optimizers
- 用高阶龙格-库塔法改进Adam，但实际等同于固定步长的平均梯度法。
- 在相同计算量下，训练损失反而高于普通Adam，泛化能力未提升。
- 真正有效的是步长自适应机制，但对初始值敏感且无法提升测试精度。
将优化器视为梯度流离散化，促使人们将高阶龙格-库塔（RK）积分器应用于神经网络。本文构建了一个代表性的Adam变体（Bogacki-Shampine 3(2) RK对，FSAL复用，局部误差步长控制），并在严格的计算匹配协议下评估其表现——所有方法均享有相同的梯度评估预算。在此条件下，该RK变体在小批量和全批量训练中均劣于普通Adam。仪器化分析显示，所谓的“自适应”实为虚幻：归一化误差远低于容差，步长从第一步起即被锁定在最大增长上限（98-100%的步骤），不同容差设置（跨度达100倍）产生比特级相同的轨迹。该方法本质上是代价3-4倍的固定步长Adam，使用平均梯度。修复算法（真正的拒绝分支；作用映射上的误差）逆转了全批量结果——训练损失比调优后的Adam低约40倍，而固定步长控制揭示出自适应机制（一种涌现的暖启动与增长调度）是关键。然而，该收益对初始步长敏感，且未能提升测试精度。预注册后续研究排除了明显解释：更深最小化不导致过拟合，显式温度调节仅恶化性能，最终指向轨迹效应——控制器选择的最小值比一阶下降低1.3-3.4个点，但泛化性更差。一项n=10的研究确认一个次要效果：梯度平均本身是真实隐式正则化项，在10/10种子上优于学习率匹配的Adam和AdamW，但RMSprop和NAdam以三分之一的每步成本即可匹敌或超越。高阶自适应积分带来更深的确定性最小化和微弱正则化，但这些均可由更便宜、调优得当的一阶基线实现。
原文摘要 · Abstract (English)
Interpreting optimizers as gradient-flow discretizations has motivated applying higher-order Runge-Kutta (RK) integrators to neural networks. We build a representative Adam variant (Bogacki-Shampine 3(2) RK pair, FSAL reuse, local-error step control) and evaluate it under a strict compute-matched protocol giving every method the same gradient-evaluation budget - an accounting this literature rarely enforces. Under it the RK variant loses to plain Adam on training loss in both minibatch and full-batch (RK's best-case) training. Instrumenting it shows the "adaptivity" is illusory: normalized error stays far below tolerance, the step size pins at its growth cap from step one (98-100 percent of steps), and no rtol x hmax x h0 setting makes it act; tolerances spanning 100x give bit-identical trajectories. The method is exactly fixed-step Adam with an averaged gradient at 3-4x cost. Repairing it (true reject branch; error on the applied map) reverses the full-batch result - about 40x lower training loss than tuned Adam - and a fixed-step control isolates adaptivity (an emergent warmup-and-growth schedule) as the mechanism. But the gain is fragile to the initial step size and does not reach test accuracy. A pre-registered follow-up rules out the obvious explanations: deeper minimization does not overfit, and an explicit temperature knob only hurts - leaving a trajectory effect, the controller selecting a minimum generalizing 1.3-3.4 points below first-order descent at equal depth. An n=10 study confirms one secondary effect: gradient averaging is a genuine implicit regularizer, beating lr-matched Adam and AdamW on 10/10 seeds - yet RMSprop and NAdam match or beat it at a third the per-step cost. Higher-order adaptive integration buys deeper deterministic minimization and a small regularization effect, but nothing a cheaper, well-tuned first-order baseline does not already provide.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。