arXiv:2605.26037cs.CL2026-05中稿 · EMNLP

模型使用知识图谱时出现先升后降的性能崩溃,根源在接口反馈缺失。

Peak-Then-Collapse and the Four Interface Channels of Knowledge-Graph Tool Use

  • 用最小化知识图谱接口测试强化学习,发现性能先升后崩
  • 失败模式共性:改奖励也难根治,因接口无自然语言反馈
  • 自蒸馏可稳定提升至40%准确率,且不依赖模型大小

我们在一个极简的知识图谱工具API上测试标准强化学习工具使用方法(基于Qwen2.5-7B-Instruct的GRPO),仅包含四个Freebase导航动词和Complex WebQuestions数据集。在自验证检索奖励下,策略的工具导向回答率从3.8%升至9.6%(250步),随后在50步内骤降至0%,呈现重复出现的“峰后坍塌”现象。七种奖励设计均显示四种典型失败模式,增加更密集或精准的代理奖励仅转移而非消除失败。我们指出,与Python解释器、网页搜索和JSON API不同,该接口缺乏预训练中见过的语言信号反馈——空结果[]不提供错误线索。移除表面特征后暴露出一种系统性退化,同类奖励重设计无法解决。直接使用真值关系的消融实验表明,注入金标关系仅提升0.20个百分点精确匹配率,95.4%的错误源于检索组合而非答案提取。作为缓解方案,单轮自蒸馏在7B模型上达到40.0% EM,且容量不变:14B模型仅提升0.25个百分点,初始化影响微弱,表明上限受接口限制。

原文摘要 · Abstract (English)

We test the standard RLVR tool-use recipe -- GRPO on Qwen2.5-7B-Instruct -- on a deliberately minimal knowledge-graph tool API: four Freebase navigation verbs over Complex WebQuestions. Under a self-verifiable retrieval reward, the policy's tool-grounded answer rate climbs from $3.8\%$ to $9.6\%$ over 250 steps, then collapses to $0\%$ within a single 50-step window -- a \emph{peak-then-collapse} pattern replicated across four seeds. Across seven reward designs, we find four recurring failure modes: adding denser or more targeted proxy rewards shifts the failure mode rather than eliminating it. We argue that a key difference from Python interpreters, web search, and JSON APIs is interface feedback: their failures often leak natural-language signal the model saw in pretraining. A Python traceback names the failing line; an empty Freebase result \texttt{[]} does not. Stripping away that surface exposes a degradation regime that same-family reward redesigns do not fix. A direct oracle ablation rules out relation selection: injecting gold relations at every retrieval call lifts exact-match accuracy by only $+0.20$~pp, and $95.4\%$ of retrieval-dependent errors are retrieval-composition failures rather than answer-extraction failures. As a mitigation, one-iteration self-distillation reaches $40.0\%$ EM at 7B and is capacity-invariant: doubling capacity to 14B improves EM by only $0.25$~pp, and initialization barely matters -- the ceiling appears interface-bound within the 7B--14B range tested.

知识图谱强化学习工具使用接口设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。