给长序列任务中的每一步精准分配奖励,提升智能体学习稳定性。
SALT: Step-level Advantage Assignment for Long-horizon Agents via Trajectory Graph
- 构建轨迹图量化每一步质量,实现细粒度奖励分配。
- 在多个基准上提升性能,最大提升达18.7%。
- 可无缝接入现有算法,计算开销极低,适合长程任务研究者。
大型语言模型(LLMs)在单轮任务中表现优异,但在复杂、多步骤的长时序任务中仍面临挑战。主流强化学习方法依赖稀疏的结果奖励,尤其在无价值函数的群体型算法(如GRPO)中,对整个轨迹统一奖惩会导致训练不稳定和策略不佳,因有益与有害动作常混杂于多步交互中。为此,我们提出SALT,一种轻量级框架,仅基于最终结果奖励,通过构建相同提示下的轨迹图,量化每一步的质量并精确分配优势值。SALT设计为即插即用模块,无需修改采样流程,计算开销可忽略。在WebShop、ALFWorld和AppWorld多个基准上,不同规模模型的实验表明,SALT持续提升性能,最大提升达18.7%。我们还深入分析了设计选择,提供可操作的优化建议。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities, enabling language agents to excel at single-turn tasks. However, their application to complex, multi-step, and long-horizon tasks remains challenging. While reinforcement learning (RL) offers a promising avenue for addressing these challenges, mainstream approaches typically rely solely on sparse, outcome-based rewards, a limitation that becomes especially problematic for group-based RL algorithms lacking critic models, such as Group Relative Policy Optimization (GRPO). In such methods, uniformly rewarding or penalizing all actions within a trajectory can lead to training instability and suboptimal policies, because beneficial and detrimental actions are often entangled across multi-step interactions. To address this challenge, we propose SALT, a novel and lightweight framework that provides a finer-grained advantage assignment, derived solely from outcome rewards. We achieve this by constructing a graph from trajectories of the same prompt, which allows us to quantify the quality of each step and assign advantages accordingly. Crucially, SALT is designed as a plug-and-play module that seamlessly integrates with existing group-based RL algorithms, requiring no modifications to the rollout procedure and introducing negligible computational overhead. Extensive experiments on the WebShop, ALFWorld, and AppWorld benchmarks with various model sizes demonstrate that SALT consistently improves performance. We also conduct a thorough analysis to validate the design choices behind SALT and offer actionable insights.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。