用AI自动发现更好的语言模型优化算法，提升推理能力。
From AI Assistant to AI Scientist: Autonomous Discovery of LLM-RL Algorithms with LLM Agents
- 构建闭环系统POISE，通过结构化档案迭代探索算法机制。
- 从GRPO出发，发现新算法使数学推理得分提升4.6分，通过率升至43.3%。
- 适合对强化学习与大模型优化感兴趣的研究者使用。
语言模型策略优化算法的改进仍依赖耗时的人工设计与验证。不同于简单的代码组合搜索，该问题需在训练动态紧密耦合的机制空间中探索，并复用多轮实验证据。我们提出POISE——一种面向语言模型策略优化算法的闭环自动化发现框架。该框架维护一个结构化、谱系关联的档案库，包含算法提案、可执行实现、标准化评估及自然语言反思，支持基于证据的迭代。以GRPO为起点，在数学推理实验中评估64个候选算法，发现包括解析方差缩放和有效性掩码在内的改进机制。最优变体将加权总分从47.8提升至52.5（+4.6），并使AIME25 pass@32从26.7%增至43.3%，验证了自动化策略优化发现的可行性，同时支持可解释的设计原则。
原文摘要 · Abstract (English)
Discovering improved policy optimization algorithms for language models remains a costly manual process requiring repeated mechanism-level modification and validation. Unlike simple combinatorial code search, this problem requires searching over algorithmic mechanisms tightly coupled with training dynamics while reusing empirical evidence across iterations. We propose POISE, a closed-loop framework for automated discovery of policy optimization algorithms for language models. POISE maintains a structured, genealogically linked archive linking proposals, executable implementations, standardized evaluations, and natural-language reflections to support evidence-driven iteration. In mathematical reasoning experiments starting from GRPO, POISE evaluates 64 candidate algorithms and discovers improved mechanisms, including analytic-variance scaling and validity masking. The best variant improves weighted Overall from 47.8 to 52.5 (+4.6) and increases AIME25 pass@32 from 26.7% to 43.3%, demonstrating the feasibility of automated policy optimization discovery while supporting interpretable design principles.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。