提出无需调参的优化算法，自动学习权重并加速收敛。
Fast Compute for ML Optimization
- 基于模型自适应权重的期望最大化更新，替代传统学习率和动量设置。
- 在弱条件逻辑回归上，性能比调优后的Adam低13倍最终损失。
- 共享统计量可使正则化路径计算提速10倍，适合高维优化场景。
我们研究具有方差-均值尺度混合表示的损失函数优化问题。在此表示下，每次EM迭代为加权最小二乘更新，潜变量决定观测值与参数权重；其作用类似于Adam的二阶矩缩放与AdamW的权重衰减，但由模型自身推导。由此提出的尺度混合EM（SM-EM）算法消除了用户指定的学习率和动量调度。在维度 $p \in \{20, \ldots, 500\}$ 的合成弱条件逻辑回归基准上，采用Nesterov加速的SM-EM相比通过学习率网格搜索调优的Adam，最终损失最低可达其1/13。对于40个正则化路径点，跨惩罚值共享充分统计量相较相同调优的Adam协议可实现10倍的运行时间降低。基础（非加速）版本保证目标函数值单调不增；加入Nesterov外推则以牺牲该保真性为代价换取更快的实证收敛速度。
原文摘要 · Abstract (English)
We study optimization for losses that admit a variance-mean scale-mixture representation. Under this representation, each EM iteration is a weighted least squares update in which latent variables determine observation and parameter weights; these play roles analogous to Adam's second-moment scaling and AdamW's weight decay, but are derived from the model. The resulting Scale Mixture EM (SM-EM) algorithm removes user-specified learning-rate and momentum schedules. On synthetic ill-conditioned logistic regression benchmarks with $p \in \{20, \ldots, 500\}$, SM-EM with Nesterov acceleration attains up to $13\times$ lower final loss than Adam tuned by learning-rate grid search. For a 40-point regularization path, sharing sufficient statistics across penalty values yields a $10\times$ runtime reduction relative to the same tuned-Adam protocol. For the base (non-accelerated) algorithm, EM monotonicity guarantees nonincreasing objective values; adding Nesterov extrapolation trades this guarantee for faster empirical convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。