通过修正奖励信号,让大模型智能体更专注长期任务中的关键步骤。
STAPO: Selective Trajectory-Aware Policy Optimization for LLM Agent Training

- 用归一化熵识别任务中偏离正常行为的异常步骤。
- 在三个基准上实现当前最优性能,显著减少任务中途遗忘现象。
- 适合需要长期规划与稳定决策的复杂智能体训练场景。
强化学习是训练大语言模型智能体完成长程任务的主要范式。然而,稀疏且延迟的奖励常导致轨迹忽视问题,即智能体在中间步骤失去对任务目标和交互历史的关注。已有工作尝试使用香农熵作为不确定性信号进行步级监督,但该方法将状态内在复杂性与智能体信心混为一谈,导致决策可靠性估计不可靠。为此,我们提出归一化熵,衡量智能体在特定状态下的信心偏差相对于其平均行为的表现,从而强化低质量动作与轨迹忽视之间的关联。基于此,我们提出分层分组式的强化学习框架——选择性轨迹感知策略优化(STAPO)。STAPO利用归一化熵定位与轨迹忽视相关的异常步骤,并通过轨迹感知奖励与轨迹无关惩罚的联合机制进行优化,既增强了轨迹感知能力,又保持了训练稳定性。在ALFWorld、WebShop和Search-Augmented QA上的大量实验表明,STAPO在显著缓解轨迹忽视的同时达到当前最优性能,验证了其在智能体任务中的有效性与鲁棒性。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) is the dominant paradigm for training Large Language Model (LLM) agents on long-horizon tasks. However, sparse and delayed rewards often lead to trajectory neglect, in which agents lose focus on the task goal and interaction history at intermediate steps. Prior work has explored step-level supervision using Shannon-entropy-based uncertainty signals, which conflate inherent state complexity with agent confidence and therefore provide unreliable estimates of decision reliability. To address this issue, we propose normalized entropy, which measures confidence deviations relative to an agent's average behavior under a given state, thereby strengthening the association between low-quality actions and trajectory neglect. Building on this insight, we introduce Selective Trajectory-Aware Policy Optimization (STAPO), a hierarchical group-based RL framework. STAPO leverages normalized entropy to locate outlier steps associated with trajectory neglect and optimizes them via a joint mechanism of trajectory-aware reward and trajectory-independent penalty, enhancing trajectory awareness while preserving training stability. Extensive experiments on ALFWorld, WebShop, and Search-Augmented QA demonstrate that STAPO achieves state-of-the-art performance while substantially alleviating trajectory neglect, validating its effectiveness and robustness for agentic tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。