arXiv:2605.24517cs.LGcs.CL2026-05被引 6

让终端代理从每轮反馈中自动学习世界模型,无需额外数据。

ECHO: Terminal Agents Learn World Models for Free

论文配图:ECHO: Terminal Agents Learn World Models for Free
图 1 · 摘自论文原文
  • 用环境输出作为辅助信号,让模型预测终端响应
  • 在终端基准上使成功率翻倍,8B版从2.70%升至5.17%
  • 无需专家数据即可自提升,适合想优化命令生成的开发者

CLI代理是语言模型最接近具身化的设置:模型发出命令,终端执行并返回输出流(标准输出、错误、文件、日志和追踪记录),这些结果构成环境反馈。我们提出,这一流本身就是监督信号,但传统强化学习会忽略它:GRPO类训练仅用稀疏的最终奖励更新动作,丢弃了回放中已有的环境响应。失败回放虽含丰富信息却难以提供策略梯度信号。为此,我们提出ECHO(环境交叉熵混合目标),将标准策略梯度损失与辅助损失结合,训练策略预测自身动作引发的环境观察令牌。ECHO复用原有前向传播,无需额外回放,将终端反馈转化为所有回放的密集监督信号。在TerminalBench-2.0上,其使Qwen3-8B的pass@1从2.70%提升至5.17%,Qwen3-14B从5.17%提升至10.79%。且在未见轨迹上显著降低环境令牌交叉熵,而GRPO几乎无变化。从基础Qwen3-8B出发,ECHO在未见任务上达到专家微调+GRPO的性能,恢复约一半专家初始化收益。某些场景下,仅靠环境预测损失即可实现无需验证器的自我改进,使模型仅通过环境交互就提升对分布外任务的适应能力。结果表明,环境观察不仅是未来动作的上下文,更是每轮回放中已存在的密集、在线监督信号。

原文摘要 · Abstract (English)

CLI agents are the closest thing language models have to an embodied setting: the model emits commands, the terminal executes them, and the returned stream -- stdout, errors, files, logs, and traces -- records the consequences. We argue that this stream is a supervision signal, but standard agent RL discards it: GRPO-style training updates action tokens with sparse outcome-level rewards while ignoring environment responses already in the rollout. Failed rollouts provide little policy-gradient signal despite containing rich evidence about how the environment responds. We introduce ECHO (Environment Cross-entropy Hybrid Objective), a hybrid objective that combines the standard policy-gradient loss on action tokens with an auxiliary loss that trains the policy to predict environment observation tokens resulting from its own actions. ECHO reuses the same forward pass as GRPO, requires no additional rollouts, and turns terminal feedback into dense supervision for all rollouts. ECHO doubles GRPO pass@1 on TerminalBench-2.0: Qwen3-8B improves from 2.70% to 5.17%, and Qwen3-14B from 5.17% to 10.79%. ECHO also produces policies that better predict terminal dynamics, even on trajectories they did not generate: across held-out rollouts, it sharply reduces environment-token cross-entropy while GRPO alone barely changes it. From base Qwen3-8B, ECHO matches expert-SFT-then-GRPO performance on held-out terminal tasks without expert demonstrations, and recovers roughly half of the expert-SFT initialization benefit on TerminalBench-2.0. In some settings, the environment prediction loss alone enables verifier-free self-improvement, allowing policies to improve on unseen OOD tasks by learning only from environment interactions. Together, these results suggest that environment observations are not merely context for future actions, but a dense, on-policy supervision signal already present in every rollout.

智能代理世界模型强化学习终端生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。