arXiv:2607.27652cs.CL2026-07

重构检索接口,让搜索智能体更精准地选择证据。

Harness-G: A Graph-Structured Harness for Search Agents

论文配图:Harness-G: A Graph-Structured Harness for Search Agents
图 1 · 摘自论文原文
  • 将自由文本查询改为有限动作选择,减少语言歧义。
  • 在六项问答任务中,1.5B模型比最强基线高10.74分。
  • 适合研究检索优化与强化学习智能体的学者。

强化学习搜索智能体通常将检索建模为自由形式的自然语言查询生成,并使用最终答案奖励优化多轮交互。现有研究主要通过更密集或更结构化的信用信号提升训练效果,但很少审视检索在策略-环境接口上的建模是否合理。我们观察到在Search-R1训练中存在显著的检索混淆现象:相同问题的不同推演路径持续生成不同的查询字符串,但累积证据集却日益重叠。我们称此现象为检索等价性崩溃;在此状态下,轨迹在检索决策上趋于效用等价,组内回报差异微弱。为此,我们提出Harness-G,一种图结构的检索框架,重新设计该接口。它将自由形式的查询生成转化为有限动作选择:策略选择一个证据句或实体,或直接作答,而环境负责构建菜单、跟踪检索状态,并验证和执行每项选择。该接口减少了语言歧义,使同状态下的备选动作可直接比较。基于此接口,我们引入结构化非短视信用(SNC),利用冻结的答案评分器比较所选动作与其替代动作,并将下游收益分配给促成这些动作的早期决策。在六个QA基准测试中,Harness-G在两个评估模型规模下均取得最高平均F1,1.5B规模下比最强基线Graph-R1高出10.74分,3B规模下高出3.98分。

原文摘要 · Abstract (English)

Reinforcement learning (RL) search agents commonly model retrieval as free-form natural-language query generation and optimize multi-turn interactions using final-answer rewards. Current studies mainly improve training with denser or more structured credit signals, but rarely examine whether retrieval is properly formulated at the policy-environment interface. We observe pronounced retrieval aliasing during Search-R1 training: rollouts for the same question continue to generate distinct query strings, yet their accumulated evidence sets increasingly overlap. We call this phenomenon retrieval-equivalence collapse; in this regime, trajectories approach utility equivalence with respect to retrieval decisions, leaving within-group returns with little effective retrieval contrast. To address this problem, we propose Harness-G, a graph-structured retrieval framework that redesigns this interface. It reformulates free-form query generation as finite action selection: the policy selects an evidence sentence or entity, or chooses to answer, while the environment constructs the menu, tracks retrieval state, and validates and executes each choice. This interface reduces linguistic aliasing and makes same-state alternatives directly comparable. Building on this interface, we introduce Structured Non-myopic Credit (SNC), which uses a frozen answer scorer to compare the selected action with its alternatives and assigns downstream gains to the earlier actions that enabled them. Across six QA benchmarks, Harness-G achieves the highest average F1 at both evaluated model scales, outperforming the strongest baseline, Graph-R1, by 10.74 points at 1.5B and 3.98 points at 3B.

强化学习搜索智能体检索优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。