arXiv:2607.19616cs.LG2026-07

KG结构对强化学习的帮助取决于任务和注入方式,不是所有情况下都有用。

The Mechanism Matters: When Knowledge Graphs Help Reinforcement Learning

论文配图:The Mechanism Matters: When Knowledge Graphs Help Reinforcement Learning
图 1 · 摘自论文原文
  • 通过控制实验,验证了知识图谱的结构比随机特征更有效。
  • 在组合稀疏奖励任务中,正确结构的KG使成功率从70%提升至97%。
  • 硬性掩码易受错误信息影响,软性注入则能容忍错误知识。

知识图谱(KG)被广泛用于向强化学习(RL)注入先验知识,但现有研究多为单一领域、正向结果论文,缺乏对何时KG有效、中立或有害的系统分析。本研究在可控的合成知识图谱上,独立变化强化学习任务、注入机制(状态特征、动作掩码、基于势能的奖励设计)与KG质量。使用覆盖MiniGrid环境的可控合成知识图谱,发现:第一,在组合稀疏奖励任务中,结构化引导显著提升样本效率与求解可靠性(70%→97%成功种子),而仅打乱边位置的对照组失去优势(掩码p=0.0001;奖励设计p=0.006),说明收益来自结构而非通用正则化。第二,KG价值随任务相关知识量增加而上升。第三,最关键的发现是安全性依赖于机制:软性、保优注入能利用正确知识并忽略错误知识,而硬性掩码在知识不完整或错误时会禁用必要动作,使错误知识比无知识更糟。基于UMLS的败血症管理离线强化学习案例为零结果,强调效益需任务结构与机制匹配。研究为实践者提供具体指导,明确何时及多大程度可信任知识图谱。

原文摘要 · Abstract (English)

Knowledge graphs (KGs) are widely used to inject prior knowledge into reinforcement learning (RL), yet the literature is dominated by single-domain, positive-result method papers, so we lack a systematic account of when KG structure helps an agent, when it is neutral, and when it hurts. We conduct a controlled study that independently varies the RL task, the injection mechanism (state features, action masking, or potential-based reward shaping), and KG quality. Using a synthetic, fully controllable KG over MiniGrid environments, we report three findings. First, on compositional sparse-reward tasks structured KG guidance improves sample efficiency and solve reliability (70% to 97% of seeds), and a shuffle control that permutes the KG's edges while preserving their count collapses the benefit toward baseline (masking p=0.0001; shaping p=0.006), so the gain is structural rather than generic regularization. Second, KG value scales with the amount of task-relevant knowledge the graph contains. Third, and most consequential, safety depends on the mechanism: soft, optimality-preserving injection benefits from correct knowledge and harmlessly ignores incorrect knowledge, whereas hard masking is brittle, forbidding essential actions when the KG is incomplete or corrupted and making a wrong KG worse than none. A UMLS-derived clinical case study on sepsis management under offline RL is a careful null, underscoring that benefits require task structure the chosen mechanism can exploit. Our results give practitioners concrete guidance on how, and how much, to trust a KG when using it to guide RL.

强化学习知识图谱结构依赖注入机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。