arXiv:2608.18489cs.CL2026-08

提出诊断框架MissDiag,精准定位KGQA系统在知识缺失时的脆弱点。

MissDiag: Diagnostic Evaluation of Incomplete-Knowledge Robustness in KGQA and KG-RAG

论文配图:MissDiag: Diagnostic Evaluation of Incomplete-Knowledge Robustness in KGQA and KG-RAG
图 1 · 摘自论文原文
  • 通过结构化缺失干预,分解鲁棒性下降的来源
  • 发现答案邻近证据丢失导致最大性能下降
  • 适合评估和调试知识图谱问答系统的缺陷

知识图谱问答(KGQA)与基于知识图谱的检索增强生成(KG-RAG)旨在基于显式图谱证据回答问题,但现实中的知识图谱常存在稀疏、过时和不完整问题。现有鲁棒性评估通常仅报告证据被移除或扰动后的整体答案质量变化,无法区分退化原因:同一分数下降可能由缺失证据类型、系统响应差异或答案匹配协议敏感性共同导致。为此,我们提出MissDiag——一种针对KGQA与KG-RAG在不完整知识下的诊断评估框架。该框架在保持问题与正确答案不变的前提下,对基准提供的支持图施加结构化的缺失干预,实现成对比较,从而将鲁棒性变化分解为证据类型、系统响应与评估协议三个维度,而非简化为单一分数降幅。跨多个系统族的实验表明,不完整知识下的鲁棒性应被视为类型化退化现象而非统一属性:答案邻近证据丢失引发最大性能下降;源上下文移除常无影响甚至有益;语义答案匹配改变绝对得分但维持主要类型化退化模式。通过将聚合测量转化为类型化诊断归因,MissDiag为比较、诊断和压力测试KGQA与KG-RAG系统在知识不全条件下的表现提供了更可解释的基础。

原文摘要 · Abstract (English)

Knowledge graph question answering (KGQA) and knowledge-graph-based retrieval-augmented generation (KG-RAG) aim to ground answers in explicit graph evidence, but real-world knowledge graphs are often sparse, outdated, and incomplete. Existing robustness evaluations usually report aggregate changes in answer quality after evidence is removed or perturbed, which measures sensitivity to incomplete support but leaves the source of degradation under-specified: the same score change can conflate the type of missing evidence, the response of the evaluated system, and the sensitivity of the answer-matching protocol. To address this gap, we propose \textbf{MissDiag}, a diagnostic evaluation framework for incomplete-knowledge robustness in KGQA and KG-RAG. MissDiag keeps the question and gold answer fixed while applying structurally typed missingness interventions to benchmark-provided support graphs, enabling paired comparisons that decompose robustness changes by evidence type, system response, and evaluation protocol rather than reducing them to a single aggregate score drop. Experiments across multiple system families show that incomplete-knowledge robustness is better understood as a typed degradation phenomenon than as a uniform property: answer-adjacent evidence loss produces the largest observed degradation, source-context removal is often neutral and can be beneficial, and semantic answer matching changes absolute scores while preserving the main typed degradation patterns. By transforming aggregate robustness measurement into typed diagnostic attribution, MissDiag provides a more interpretable basis for comparing, diagnosing, and stress-testing KGQA and KG-RAG systems under incomplete knowledge.

知识图谱鲁棒性评估诊断分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。