arXiv:2608.04934cs.CLcs.LG2026-08被引 1

让大模型智能体通过环境探索自动学习任务,无需人工设计任务或标注。

State2State: Environment-Derived Mid-Training for LLM Agents

论文配图:State2State: Environment-Derived Mid-Training for LLM Agents
图 1 · 摘自论文原文
  • 从环境探索中自动生成训练目标,用状态匹配验证完成度。
  • 在ALFWorld和ScienceWorld上提升智能体性能,跨环境泛化效果好。
  • 适合想降低人工干预、实现大规模自主训练的研究者。

传统大模型智能体训练依赖专家轨迹的监督微调或人工设计任务的在线强化学习,受限于外部指定的任务与监督信号,难以扩展和多样化。本文提出一种环境驱动的中段训练方法State2State,通过将探索到的环境状态转化为训练目标,让智能体挑战到达指定目标状态。该方法利用规则匹配验证任务完成,无需专家监督或手动任务设计,可实现可扩展、可验证的训练。在ALFWorld和ScienceWorld上的实验表明,State2State作为独立训练阶段可普遍提升智能体表现;作为下游强化学习的初始化,进一步提升最终性能与学习效率,并展现出有前景的跨环境泛化能力。

原文摘要 · Abstract (English)

Training LLM agents commonly relies on supervised fine-tuning from expert trajectories or online reinforcement learning over human-specified tasks with handcrafted verifiers. Though effective, both remain bottlenecked by externally specified tasks and supervision signals, limiting the scalability and diversity of agent training. We study an environment learning paradigm in which agents acquire interaction and manipulation capabilities solely through environment interaction, without externally specified tasks. We propose State2State, an environment-derived mid-training method that converts explored environment states into training objectives, challenging agents to reach a specified target state. By deriving tasks from environment exploration and verifying success through rule-based state matching, State2State provides scalable and verifiable training objectives without expert supervision or manual task design. Experiments on ALFWorld and ScienceWorld show that State2State improves agent performance as a standalone environment-learning stage in most settings. As initialization for downstream RL, it further improves final performance and learning efficiency, with promising evidence of cross-environment generalization.

智能体训练自监督环境学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。