arXiv:2606.17682cs.CL2026-06被引 1

用大模型自动设计强化学习训练环境,提升训练效率。

From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning

论文配图:From Trainee to Trainer: LLM-Designed Training Environment for RL with Multi-Agent Reasoning
图 1 · 摘自论文原文
  • 让大模型分析失败轨迹,自动优化下一阶段训练环境配置。
  • 在多智能体测试平台中表现优于更大商用模型和固定环境基线。
  • 训练过的模型比原始模型更懂如何改进自身训练环境。

大型语言模型(LLM)训练中的强化学习流程通常依赖人工重设计训练环境,需从业者凭经验推测最优配置。为此,我们提出 LLM-as-Environment-Engineer 框架:当前策略模型结合失败轨迹与上下文信息,生成下一阶段环境配置。我们还引入 MAPF-FrozenLake,一个可调控的测试平台,其生成器支持多维环境配置,适用于环境重设计的研究与评测。在该平台上,环境工程师基于策略行为摘要、失败案例和环境统计等结构化信息生成新配置。以 Qwen3-4B 为骨干,本框架在基准测试中取得最强综合性能,超越更大规模的专有模型(如 GPT、Gemini)及固定环境训练基线。进一步分析发现,有效环境更新依赖失败证据,并保留已有效配置。有趣的是,当前强化学习检查点作为环境工程师的表现优于原始基础模型,表明策略学习提升了模型诊断自身缺陷的能力。

原文摘要 · Abstract (English)

Reinforcement learning pipelines for Large Language Model (LLM) training often rely on manually redesigned environments between stages, requiring practitioners to heuristically infer which configuration will best improve the current policy. To automate this process, we propose the LLM-as-Environment-Engineer framework in which the current policy model analyzes failure trajectories together with contextual information and proposes modifications to the next-stage training environment configuration. We also introduce MAPF-FrozenLake, a controllable testbed whose generator exposes multi-dimensional environment configurations, making it suitable for studying and benchmarking environment redesign. On this testbed, we condition the environment engineer on structured summaries of policy behavior, failure cases, and environment statistics, from which it produces the configuration for the next training stage. With Qwen3-4B as the backbone, our framework achieves the strongest aggregate performance on our benchmarks, outperforming larger proprietary LLMs (e.g., GPT, Gemini) and fixed-environment training baselines. We further analyze which forms of context are most effective, finding that successful environment updates rely on failure evidence and preserve configurations that already work. Interestingly, the current RL checkpoint serves as a better environment engineer than the original base model, suggesting that policy learning improves the model's ability to diagnose its remaining weaknesses.

强化学习大模型训练环境设计多智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。