用多智能体框架自动发现高效优化器，提升Transformer预训练效果。
OPTScientist: Multi-Agent Discovery of Typed Optimizer Programs for Transformer Pretraining
- 四类智能体协作设计带类型约束的优化器程序。
- 发现的新优化器RS-MR在多个任务上超越强基线模型。
- 适合对自动化机器学习和优化器设计感兴趣的研究者。
为现代深度学习设计优化器仍是一个具有挑战性的科学问题，需综合考虑优化几何、状态动态、数值稳定性、实现约束及经验泛化能力。现有自动化优化器发现方法通常在无约束代码空间或狭窄参数化优化器族中搜索，前者灵活但常生成无效或不可解释的程序，后者稳定但限制创新。我们提出OPTScientist，一种基于理论引导的多智能体框架，用于在带类型的领域特定语言（DSL）中进行优化器发现。该框架将优化器设计视为受约束的科学搜索过程，候选更新通过方向、缩放、预条件、正则化、状态和分组模块表达。四个角色智能体——理论家、设计者、工程师和评审员——在单一编排循环中协作，提出假设、合成DSL候选、编译评估优化器并批评结果。为克服固定搜索空间的局限，当重复失败揭示表示瓶颈时，系统会引入第二阶段机制，提出微小的DSL扩展。利用此框架，我们发现了RS-MR——一种低状态矩阵优化器，在本研究的原生评估协议下，其在Transformer预训练中优于强基线。结果表明，一条通往基于理论、类型化程序、编译验证与闭环实验的自动化优化器科学之路。
原文摘要 · Abstract (English)
Designing optimizers for modern deep learning remains a challenging scientific problem, requiring the joint consideration of optimization geometry, state dynamics, numerical stability, implementation constraints, and empirical generalization. Existing automated optimizer discovery methods typically search either over unconstrained code spaces or within narrowly parameterized optimizer families. The former is flexible but often produces invalid or uninterpretable programs, while the latter is stable but limits novelty. We introduce OPTScientist, a theory-guided multi-agent framework for optimizer discovery in a typed domain-specific language (DSL). OPTScientist formulates optimizer design as a constrained scientific search process, where candidate updates are expressed through direction, scaling, preconditioning, regularization, state, and grouping modules. Four role agents, Theorist, Designer, Engineer, and Reviewer, collaborate within a single orchestration loop to propose hypotheses, synthesize DSL candidates, compile and evaluate optimizers, and critique results. To overcome the limitations of a fixed search space, OPTScientist combines evolutionary search over optimizer programs with a second-stage mechanism that proposes small DSL extensions when repeated failures reveal representational bottlenecks. Using this framework, we discover RS-MR, a reduced-state matrix optimizer that improves transformer pretraining over strong baselines under our native evaluation protocol. Our results suggest a path toward automated optimizer science grounded in theory, typed programs, compiler validation, and closed-loop experimentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。