arXiv:2607.04033cs.LGcs.AI2026-07

系统梳理上百种优化器,构建跨领域评测体系。

OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers

论文配图:OmniOpt: Taxonomy, Geometry, and Benchmarking of Modern Optimizers
图 1 · 摘自论文原文
  • 将优化器更新拆解为五阶段流程,揭示多数方法仅用一两个阶段。
  • 设计统一评测框架,覆盖从语言模型到图像分类的多种训练场景。
  • 提供可操作的选型坐标系,适合模型训练系统设计者参考。

大规模模型训练中的优化器选择已成为受计算、内存、调参预算和任务多样性共同约束的系统级决策问题,而现有超过一百种方法的生态仍零散割裂。为此,我们提出OmniOpt,一个面向研究社区的优化器统一综述与基准手册。OmniOpt基于四个相互关联的组件:首先,将每个优化器更新视为通过五阶段元流程的结构化变换,发现多数方法仅参与其中一或两个阶段;其次,采用范数约束的线性最小化预言机(LMO)统一不同优化器;第三,这两个视角构建双维度分类体系,一个维度归类方法机制家族,另一个记录其目标优化的可度量训练指标;第四,核心部分是构建覆盖代表性优化器、模型规模与训练范式(从语言模型预训练到图像分类)的统一跨域基准,系统分析各方法族在多重效果目标下的表现,并揭示其权衡关系。OmniOpt为研究社区提供了一套在明确机制与目标假设下的优化器选型操作坐标系,并指明了优化器领域未来发展的方向。

原文摘要 · Abstract (English)

Optimizer selection for large-scale model training has become a system-level design decision constrained jointly by compute, memory, tuning budget, and task diversity, yet the landscape of over one hundred methods remains fragmented. We therefore present OmniOpt, a unified survey and benchmark cookbook of optimizers for the research community. OmniOpt rests on four coupled components. First, we treat every optimizer update as a structured transformation through a five-stage meta-pipeline, and show that most methods engage only one or two of these stages. Second, we use norm-constrained linear minimization oracles (LMOs) to unify different optimizers. Third, these two views ground a dual-dimension taxonomy, one dimension assigning each method to a mechanism family and the other recording the measurable training objectives it aims to improve. Fourth, and at the core of this paper, we instantiate the full taxonomy in a unified cross-domain benchmark spanning representative optimizers, model scales, and training regimes from language model pretraining to image classification, systematically analyzing each method family across multiple effect objectives and laying out their trade-offs. OmniOpt thus supplies the research community with an operational coordinate system for selecting optimizers under explicit mechanism and objective assumptions, and charts a direction for the future development of the optimizer community.

优化器基准评测机器学习系统设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。