arXiv:2605.29584cs.CL2026-05被引 1

用黄金动作指导强化学习,让智能体问答更准更快。

GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering

论文配图:GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering
图 1 · 摘自论文原文
  • 用黄金动作序列对齐学生模型的探索路径,提供细粒度指导。
  • 在WebQSP、GrailQA、GraphQ上均超越现有最佳性能。
  • 适合做知识库问答的强化学习研究者与工程师参考。

强化学习(RL)天然适合智能体式知识库问答(KBQA),模型需发出可执行动作,观察知识库反馈并最终返回答案。然而,现有基于RL的KBQA系统主要依赖最终答案的稀疏奖励,导致中间动作错误缺乏有效监督。尤其在逻辑形式标注的基准测试中,虽然黄金逻辑形式可转为可执行动作序列,但当前流程仅用于初始化数据构建,未用于在线策略更新。我们提出GAPD——一种训练时的黄金动作策略蒸馏框架,通过密集的令牌级引导增强基于结果的强化学习。GAPD采用MID-ANCHOR MATCHING:将学生探索和黄金执行过程中经过的中间实体作为状态锚点,通过这些实体集合匹配学生状态与黄金状态。基于对齐后的黄金动作,当前策略作为停止梯度教师,将其令牌分布蒸馏回学生策略的生成动作片段上。GAPD在WebQSP、GrailQA和GraphQ上持续超越当前最优水平。

原文摘要 · Abstract (English)

Reinforcement learning (RL) is a natural fit for agentic knowledge base question answering (KBQA), where a model must issue executable actions, observe knowledge-base feedback, and eventually return an answer. However, current RL-based KBQA systems mainly optimize sparse rewards from the final answer, leaving intermediate action errors weakly supervised. This is especially limiting for logical-form annotated KBQA benchmarks: gold logical forms can be converted into executable action sequences, but existing pipelines use them mainly for warm-start data construction rather than for on-policy RL updates. We propose GAPD, a training-time Gold-Action Policy Distillation framework that adds dense token-level guidance to outcome-based RL. To align gold actions with on-policy student rollouts, GAPD uses MID-ANCHOR MATCHING: it treats the intermediate entities reached during student exploration and gold execution as state anchors, and matches student states to gold states through these explored entity sets. The current policy conditioned on this aligned gold action serves as a stop-gradient teacher, whose token distribution is distilled back to the ordinary student policy over generated action-token spans. GAPD consistently surpasses the current state of the art on WebQSP, GrailQA, and GraphQ.

强化学习知识库问答策略蒸馏智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。