利用强化学习中的动作-观察数据，让语言智能体同时学习策略与环境模型。
Policy and World Modeling Co-Training for Language Agents
- 在强化学习过程中同步训练策略与环境模型，无需额外模拟器。
- 在三个任务基准上均超越强基线，提升稳定且显著。
- 适合想用现有强化学习流程提升智能体理解能力的研究者。
强化学习（RL）通过奖励信号指导大语言模型（LLM）智能体选择高回报动作，但对动作如何影响环境缺乏监督。世界建模（WM）可弥补这一缺陷，但现有方法常需独立模拟器、额外训练阶段或推理时额外计算。我们发现，基于策略的强化学习采样轨迹中已包含所需信号：每个转移对都关联一个动作及其产生的下一观测。基于此，我们提出PaW框架，将辅助的世界建模监督加入同一策略的强化学习训练中，不改变推理范式。为使辅助的WM监督更具信息量和稳定性，PaW引入三个组件：基于动作熵的WM数据选择、抗噪声的WM损失函数、以及奖励自适应的损失平衡机制。在三个智能体任务基准上的实验表明，该方法在不同模型和强化学习算法下均持续优于强基线。结果表明，标准强化学习采样轨迹是语言智能体训练中实用的世界建模监督来源。
原文摘要 · Abstract (English)
Reinforcement learning (RL) improves large language model (LLM) agents by teaching them which actions lead to high rewards, but provides little supervision on what those actions do to the environment. World modeling (WM) can fill this gap, yet existing approaches often require separate simulators, extra training stages, or additional inference-time computation. We observe that on-policy RL rollouts already contain the needed signal: each transition pairs an action with its resulting next observation. Based on this observation, we propose PaW, a Policy and World modeling co-training framework that adds auxiliary WM supervision to the same policy during RL, without changing the inference paradigm. To make auxiliary WM supervision informative and stable, PaW introduces three components: action-entropy-based WM data selection, noise-tolerant WM loss, and reward-adaptive loss balancing. Experiments on three agentic task benchmarks show consistent improvements over strong RL baselines across models and RL algorithms. These results suggest that standard RL rollouts are a practical source of WM supervision for language-agent training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。