无需成功轨迹,通过三重评分机制实现长任务中每一步的精准奖励。
TRCA: Transition-wise Rubric Credit Assignment for Long-horizon LLM Agents

- 基于动作触发的转换,用三类评分判断每步表现。
- 在多个任务上提升1.9%-18.3%的平均得分,效果稳定。
- 适合缺乏成功样本的早期强化学习阶段,尤其适用于复杂长序列任务。
长时序大语言模型(LLM)智能体通常依赖稀疏的终端结果进行优化,导致多步交互中的细粒度信用分配困难。现有方法或依赖过程评估器(增加标注与推理成本),或基于成功轨迹推导步骤级信用,但在强化学习初期成功轨迹极为稀少,削弱了锚点方法的有效性。本文提出过渡式评分信用分配(TRCA),直接从动作引发的转换中提取步骤级监督,无需训练评估器或成功锚点。TRCA使用证据、执行与无效性三类评分,捕捉任务相关信息获取、有效任务执行及无效或倒退行为。由此生成的基础评分衡量局部转换质量,突破性评分追踪新覆盖的证据与执行条件,奖励任务进展。结合终端结果,这些信号为策略优化提供精细的步骤级优势。在ALFWorld、WebShop及七个搜索增强型问答基准上的实验表明,相比基线方法持续提升性能。使用Qwen2.5-7B-Instruct时,WebShop得分提升6.0%-12.6%;使用Qwen2.5-3B-Instruct时,平均SearchQA得分提升1.9%-18.3%。结果证明过渡式评分信用分配对稀疏成功锚点的长时序任务具有显著有效性。
原文摘要 · Abstract (English)
Long-horizon large language model (LLM) agents are typically optimized with sparse terminal outcomes, making fine-grained credit assignment across multi-step interactions difficult. Existing approaches either rely on process evaluators, which incur annotation and inference costs, or derive step-level credit from successful trajectories. However, successful trajectories are extremely scarce during early-stage reinforcement learning, substantially weakening anchor-based methods. We propose Transition-wise Rubric Credit Assignment (TRCA), which derives step-level supervision directly from action-induced transitions without learned evaluators or successful anchors. TRCA evaluates each transition using Evidence, Execution, and Invalidity rubrics to capture task-relevant information acquisition, valid task execution, and invalid or regressive behavior. From these judgments, Foundational Rubric Reward measures local transition quality, while Breakthrough Rubric Reward tracks newly covered Evidence and Execution conditions to reward incremental task progress. Combined with terminal outcomes, these signals produce fine-grained step-level advantages for policy optimization. Experiments on ALFWorld, WebShop, and seven search-augmented question-answering benchmarks show consistent improvements over the evaluated baselines. With Qwen2.5-7B-Instruct, TRCA improves the WebShop score by 6.0%-12.6%; with Qwen2.5-3B-Instruct, it improves the average SearchQA score by 1.9%-18.3%. These results demonstrate the effectiveness of transition-wise rubric credit assignment for long-horizon tasks with sparse successful anchors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。