让大模型推理过程可解释：动态生成每步评分标准并随行为进化
ARCO: Adaptive Rubrics with Co-Evolution for Multi-Step LLM-Based Agents
- 为每一步生成动态自然语言评分标准，而非整体打分
- 在三个问答数据集上超越所有基线，最高提升12.3%准确率
- 适合研究可解释性强化学习与多步推理的开发者
多步大模型代理的强化学习通常依赖标量奖励，仅表明成功与否，无法解释优劣原因。基于评分标准的奖励虽提升了可解释性，但现有方法存在两大局限：评分在轨迹层面进行，无法指导单步行为；评分模型封闭且静态，无法随代理训练过程中行为演化而调整。本文提出ARCO（自适应评分标准协同进化），为每一步生成评分标准，并预测条件化的步骤级奖励，同时在策略回滚中持续更新评分模型，使评分标准与代理行为共同进化。在HotpotQA、2WikiMultiHopQA和MuSiQue三个数据集上，采用两种开源模型骨架，ARCO在所有设置下均优于结果、评分标准和过程奖励基线，最高提升12.3%准确率。分析表明其评分标准具备步骤特异性，对设计选择鲁棒，且有助于诊断代理行为。代码与数据已公开于https://github.com/zihangtian/ARCO。
原文摘要 · Abstract (English)
Reinforcement learning for multi-step LLM agents often relies on scalar rewards that indicate success but cannot explain why a trajectory is good or bad. Rubric-based rewards improve interpretability through natural-language criteria, but existing methods share two limitations: they score at the trajectory level, offering no guidance for individual steps; and their scorer is closed-source and static, so it cannot adapt as the agent evolves during training. We propose ARCO (Adaptive Rubric CO-evolution), which generates a per-step rubric and predicts a rubric-conditioned step-level reward for each action, and continually updates this rubric model on on-policy rollouts so that its criteria and scores co-evolve with the agent's improving behavior. Across HotpotQA, 2WikiMultiHopQA, and MuSiQue with two open-source backbones, ARCO achieves the highest EM in all settings over outcome-, rubric-, and process-reward baselines, and analyses show its rubrics are step-specific, robust to design choices, and useful for diagnosing agent behavior. Code and data are available at https://github.com/zihangtian/ARCO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。