让大模型更全面地找到多个正确答案,提升长程推理能力。
SPADER: Step-wise Peer Advantage with Diversity-Aware Exploration Rewards for Multi-Answer Question Answering

- 通过分步对比同伴返回值,精准分配每步决策的奖励。
- 在多个数据集上显著提升答案召回率和整体准确率。
- 适合需要发现稀有或多样答案的复杂问答场景。
大型语言模型正被用作工具增强型智能体,以获取超出参数化知识的信息。尽管近期研究提升了长程工具使用推理能力,但多数方法聚焦于单一正确答案的任务。而现实中许多问题需发现一组有效答案,即多答案问答(Multi-Answer QA)。该任务面临两大挑战:长搜索轨迹中的细粒度信用分配,以及对持续探索非高频实体的奖励对齐。我们提出 SPADER,一种面向多答案问答中长程工具使用的强化学习框架。SPADER 包含步骤级同伴优势(SPA),一种无需评价网络的步骤级信用分配机制,通过决策步骤对齐并行轨迹,从同伴返回值估计优势;还包含多样性感知探索奖励,通过加权稀有发现、降低冗余发现的权重,促进长尾实体发现。在 QAMPARI、Mintaka、WebQSP 和 QUEST 数据集上的实验表明,SPADER 在召回率和整体 F1 上均优于提示基代理、结果监督强化学习方法及近期步骤级监督方法。代码与模型权重已公开于 https://github.com/KhanCold/spader。
原文摘要 · Abstract (English)
Large language models are increasingly deployed as tool-augmented agents to acquire information beyond parametric knowledge. While recent work has improved long-horizon tool-use reasoning, most approaches focus on tasks with a single correct answer. In contrast, many real-world queries require discovering a comprehensive set of valid answers, a setting known as Multi-Answer QA. This setting raises two challenges: fine-grained credit assignment over long search trajectories and reward alignment for sustained exploration beyond easy high-frequency entities. We propose SPADER, a reinforcement learning framework for long-horizon tool use in Multi-Answer QA. SPADER includes Step-wise Peer Advantage (SPA), a critic-free step-level credit assignment mechanism that aligns parallel trajectories by decision step and estimates advantages from peer returns. It also includes a diversity-aware exploration reward that promotes long-tail entity discovery by upweighting rare findings and downweighting redundant ones. Experiments on QAMPARI, Mintaka, WebQSP, and QUEST show that SPADER generally improves recall and overall F1 over prompting-based agents, outcome-supervised RL methods, and recent step-level supervision approaches. Our code and model weights are available at https://github.com/KhanCold/spader.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。