arXiv:2606.16409cs.CL2026-06

让AI推理更靠谱:通过路径感知训练,避免走捷径,提升答案质量。

PathRouter: Aligning Rewards with Retrieval Quality in Agentic Graph Retrieval-Augmented Generation

论文配图:PathRouter: Aligning Rewards with Retrieval Quality in Agentic Graph Retrieval-Augmented Generation
图 1 · 摘自论文原文
  • 引入路径感知框架,区分正确答案的来源路径
  • 在3B和7B模型上分别提升平均F1 3.1和4.9
  • 适合需要可靠证据链的复杂问答任务

代理式图结构检索增强生成(Agentic GraphRAG)通过语言模型代理迭代检索与推理图结构证据,实现更准确、上下文感知的决策。然而,仅基于结果的强化学习存在“答案路径奖励混淆”问题,即正确答案可能来自捷径而非有效证据路径;同时存在“搜索-更新模糊性”,因标量轨迹反馈无法指明应调整哪些检索动作。为此,我们提出PathRouter,一种路径感知训练框架。该框架联合评估每条轨迹的答案正确性与证据路径重合度,生成四类轨迹并差异化调整优势规模,抑制捷径强化,保留寻证行为。对于证据贫乏轨迹,采用冻结的黄金证据教师模型,对推理与搜索查询词元提供逐标记KL引导,排除答案词元以避免直接响应模仿。在三个模型尺寸下六项QA基准上的实验表明,PathRouter持续提升答案F1与证据路径重合度,相较强基线平均提升3.1(3B模型)和4.9(7B模型)。

原文摘要 · Abstract (English)

Agentic GraphRAG trains language-model agents to iteratively retrieve and reason over graph-structured evidence, enabling more accurate and context-aware decision-making by efficiently navigating complex information networks. However, outcome-only reinforcement learning suffers from \textit{\textbf{answer-path reward aliasing}}, where correct answers may come from shortcuts rather than useful evidence paths. It also exhibits \textit{\textbf{search-update ambiguity}}, as scalar trajectory-level feedback does not indicate which retrieval actions to adjust. To mitigate these shortcomings, we present PathRouter, a path-aware training framework for agentic GraphRAG. PathRouter jointly evaluates each trajectory along answer correctness and evidence-path overlap, yielding four trajectory categories with differentiated GRPO advantage scaling that suppresses shortcut reinforcement while preserving evidence-seeking behavior. For evidence-poor trajectories, a frozen gold-evidence teacher provides token-level KL guidance on reasoning and search-query tokens, excluding answer tokens to avoid direct response imitation. Experiments on six QA benchmarks across three model sizes show that PathRouter consistently improves answer F1 and evidence-path overlap, achieving average F1 gains of 3.1 on 3B and 4.9 on 7B models compared to a strong baseline.

图神经网络强化学习大模型推理知识检索

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。