动态调整梯度方差衰减率，让Adam优化器更稳定应对突变梯度。
Kourkoutas-Beta: A Sunspike-Driven Adam Optimizer with Desert Flair
- 用太阳耀斑比动态调节二阶矩衰减率，自动适应梯度波动
- 在多个物理与语言任务中，相比固定beta2的Adam降低38%~58%误差
- 无需修改模型，可直接替换Adam，性能提升且计算开销极低
Transformer神经网络在物理问题建模中日益重要。数据驱动的偏微分方程（PDE）代理模型在不同边界与初始条件下训练时，常出现损失震荡和梯度尖峰；在物理信息神经网络（PINNs）中，刚性复合损失进一步加剧该现象。本文提出Kourkoutas-Beta，一种基于动态β₂的Adam风格优化器：将固定的二阶矩衰减率β₂替换为由层内梯度范数与指数移动平均比值（“sunspike”比率）驱动的动态值，压缩至[0,1)区间。尖峰时β₂降至最小值，平稳期则趋近最大值。支持漏失型AMSGrad、信任域裁剪、自适应小项及多种偏差修正模式。在四项测试中表现优异：(i) Heat2D Transformer PDE代理，(ii) Heat3D三维热传导PINN，(iii) 含抖动与罕见触发突发的小型合成任务，(iv) 30MB enwik8上的字符级Transformer（small-enwik8）。结果表明，该方法显著提升稳定性与最终损失表现；在small-enwik8上，相比Adam-0.95降低约38%、相比Adam-0.999降低约58%的bits-per-character，且方差更小。该方法为即插即用设计，在测试集A-C中运行开销与Adam相当，在D中仅增加单位数百分比。保留Adam收敛性保障的同时，大幅增强对尖峰梯度的鲁棒性。
原文摘要 · Abstract (English)
Transformer neural networks are increasingly used for physics-based problems. In data-driven PDE surrogates, training samples from varying boundary and initial conditions can cause erratic losses and spiky gradients; in physics-informed neural networks (PINNs), stiff composite losses amplify this effect. We introduce Kourkoutas-Beta, an Adam-style optimizer where the fixed second-moment discount beta2 is replaced by a layer-wise dynamic value driven by a bounded ``sunspike'' ratio: the current pooled gradient norm divided by an exponential moving average (EMA) of past norms, squashed to the interval [0,1). Spikes lower beta2 toward beta2_min; calm phases keep it near beta2_max. Options include leaky-AMSGrad (decay), trust-region clipping (max_ratio), adaptive tiny terms, and several bias-correction modes ``none'', ``beta2max'', ``exact'). With all features off and bias_correction=``none'', the method is exactly Adam. We test on four settings: (i) a Transformer PDE surrogate (Heat2D), (ii) a 3D PINN for heat conduction (Heat3D), (iii) a lightweight MLX synthetic task with jitter and rare-trigger bursts, and (iv) a character-level Transformer on 30 MB of enwik8 (small-enwik8). Kourkoutas-Beta improves stability and final loss versus fixed-beta2 Adam. On small-enwik8 it lowers bits-per-character by about 38% vs Adam-0.95 and about 58% vs Adam-0.999 over 10 seeds, with smaller variance. The method remains drop-in, with runtime overhead comparable to Adam in testbeds A-C and within single-digit percent in testbed D. It preserves Adam-style convergence guarantees while improving robustness under spiky gradients.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。