用大模型智能发现强化学习训练策略，提升性能并揭示参数演化规律
LLMZero: Discovering Adaptive Training Strategies for RL Post-Training via LLM Agents
- 让大模型代理通过树搜索自动探索训练路径，动态调整多参数
- 在4个任务上比基线提升9%至140%，优于网格搜索和随机搜索
- 发现正则化参数振荡、容量参数单调增长的通用规律，适合算法设计者
强化学习后训练策略具有数据依赖性，表现出明显规律：容量参数在各阶段单调累积，而正则化参数随训练动态主要呈现振荡。这一差异至关重要，因为固定调度无法适应非平稳的探索-利用权衡，而该原则为多阶段训练提供了可操作的设计准则。我们通过LLMZero系统实现这一发现，该系统利用大模型代理进行树搜索，诊断每个检查点的问题并提出协调的多参数转换方案。在4个不同的GRPO任务中，LLMZero发现的策略相比基线提升9%至140%（相对），比网格搜索提升6%至15%（相对），始终优于随机搜索与基于技能的代理。该结构规律可跨任务迁移，解释了为何发现策略形式各异却共享相似参数动态。
原文摘要 · Abstract (English)
RL post-training strategies are dataset-dependent and reveal a recurring empirical pattern: capacity parameters accumulate monotonically across stages, while regularization parameters predominantly oscillate in response to shifting training dynamics. This distinction matters because fixed schedules commit all parameters to fixed trajectories and therefore cannot express the non-stationary exploration-exploitation tradeoffs that regularization must track; the principle provides actionable design rules for multi-stage training. We discover this through LLMZero, a system where LLM agents search over training trajectories via tree search, diagnosing pathologies at each checkpoint and proposing coordinated multi-parameter transitions. Across 4 diverse GRPO tasks, LLMZero discovers strategies that improve over the base model by 9% to 140% relative and over grid search by 6% to 15% relative, consistently outperforming random search and the skill-based agent. The structural principle transfers across tasks, providing an explanation for why discovered strategies take qualitatively different forms yet share similar parameter dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。