对比15种优化器，发现Muon在表格数据上优于AdamW。
Benchmarking Optimizers for MLPs in Tabular Deep Learning
- 在17个数据集上系统测试15种优化器性能。
- Muon在多数任务中超越AdamW，且结果稳定。
- 适合关注训练效果的从业者和研究者。
MLP是现代表格数据深度学习架构中的常用主干网络，而AdamW是训练表格深度学习模型的默认优化器。然而，与模型结构设计不同，表格数据深度学习中优化器的选择尚未得到系统性评估，尽管新优化器在其他领域展现出潜力。为填补这一空白，我们在17个表格数据集上，基于统一实验协议，对15种优化器在训练基于MLP的模型时的表现进行了基准测试。主要发现是：Muon优化器在所有任务中均持续优于AdamW，因此应作为从业者和研究者的有力候选方案，前提是可接受其带来的训练效率开销。此外，我们发现模型权重的指数移动平均是一种简单但有效的技术，能提升原始MLP上的AdamW表现，但其效果在不同模型变体间并不一致。
原文摘要 · Abstract (English)
MLP is a heavily used backbone in modern deep learning (DL) architectures for supervised learning on tabular data, and AdamW is the go-to optimizer used to train tabular DL models. Unlike architecture design, however, the choice of optimizer for tabular DL has not been examined systematically, despite new optimizers showing promise in other domains. To fill this gap, we benchmark 15 optimizers on 17 tabular datasets for training MLP-based models in the standard supervised learning setting under a shared experiment protocol. Our main finding is that the Muon optimizer consistently outperforms AdamW, and thus should be considered a strong and practical choice for practitioners and researchers, if the associated training efficiency overhead is affordable. Additionally, we find exponential moving average of model weights to be a simple yet effective technique that improves AdamW on vanilla MLPs, though its effect is less consistent across model variants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。