让大模型智能体在训练时自动获得动态指导，提升长任务执行能力。
PATS: Policy-Aware Training Scaffolding for Agentic Reinforcement Learning
- 将技能作为动态训练支架，根据最新策略生成证据卡并调整上下文。
- 在多个任务上达到顶尖水平，同时减少25%-50%的推理令牌消耗。
- 适合需要高效长程决策的AI代理训练，尤其关注资源受限场景。
在长周期大语言模型智能体强化学习中，弱策略常重复相似错误，导致无信息量的轨迹，限制有效优化。现有以技能为中心的方法通过优化、筛选或内化可复用技能来提升探索，但其核心仍聚焦于技能本身，而非作为随策略演进而自适应的训练支持。为此，我们提出一种以策略为中心的训练范式，将技能重新构想为动态训练支架。我们的框架PATS将最新策略产生的轨迹分组转化为证据卡，并利用特定任务评估调整后续轨迹的上下文。具体指导帮助弱策略完成复杂任务。随着策略改进，冗余上下文被修订或移除，降低对显式引导的依赖，同时保留有用的轨迹多样性。策略通过标准RLVR使用环境奖励进行优化，训练支架在部署时丢弃。在ALFWorld、WebShop及七个搜索增强型问答基准上，PATS表现媲美最先进基线，同时减少25%-50%的令牌使用。
原文摘要 · Abstract (English)
In long-horizon LLM agent reinforcement learning, weak policies often repeat similar failures, producing uninformative rollout trajectories and limiting effective policy optimization. Existing skill-centric methods improve exploration by optimizing, filtering, or internalizing reusable skills. However, they remain centered on the skills themselves rather than being designed as adaptive training-time support for the evolving policy. To address this, we propose a policy-centric training paradigm that reframes skills as a dynamic training scaffold. Our framework, PATS, converts rollout groups from the latest policy into evidence cards and uses task-specific evaluation to adjust the context used in subsequent rollouts. Concrete guidance helps weak policies to complete challenging tasks. As policy improves, redundant context is revised or removed to reduce reliance on explicit guidance while preserving useful rollout variation. The policy is optimized with environmental rewards using standard RLVR, and the training scaffold is discarded at deployment. Across ALFWorld, WebShop, and seven search-augmented QA benchmarks, PATS achieves performance competitive with SOTA baselines while using 25%-50% fewer tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。