用动态知识图谱追踪对话状态,精准检测长程矛盾与不一致。
SKG-Eval: Stateful Evaluation of Multi-Turn Dialogue via Incremental Semantic Knowledge Graphs

- 构建对话演化知识图谱,显式记录实体与承诺关系。
- 在多轮对话中检测到90%以上的长期矛盾,优于现有方法。
- 输出可复现的矛盾证据,适合需要可解释评估的场景。
多轮对话评估面临挑战,因回复质量不仅取决于当前输入,还依赖先前建立的实体、主张和对话承诺。现有自动评估方法(如基于LLM或嵌入的指标)大多采用扁平化或孤立轮次表示,难以捕捉长程问题,如矛盾、话题漂移和实体不一致。为此,我们提出SKG-Eval,一种准确定、可解释的框架,将对话建模为随轮次演化的语义知识图谱(SKG),包含实体、关系和承诺。通过结构化三元组提取增量更新图谱,计算三个互补信号:(i) 局部相关性,衡量与当前提示及参考内容的一致性;(ii) 历史一致性,利用图结构与嵌入信号评估新信息与历史上下文的关联性;(iii) 逻辑连贯性,由几何矛盾引擎检测跨轮次冲突,无需NLI模型或LLM裁判。这些信号经自适应融合与加权趋势分析,生成长度无关的会话评分。在多个基准测试中,SKG-Eval与人类判断的相关性更高,显著提升对长程不一致的检测能力。此外,该框架可生成明确的矛盾证明并给出确定性得分,支持可复现、可审计的评估。结果表明,通过语义知识图谱显式追踪状态,是替代大模型隐式推理的一种可扩展方案。
原文摘要 · Abstract (English)
Evaluating multi-turn dialogue systems remains challenging because response quality depends not only on the current prompt, but also on previously established entities, claims, and conversational commitments. Existing automatic evaluators, including LLM-as-a-judge frameworks and embedding-based metrics, largely rely on flat or turn-isolated representations, making them less effective at detecting long-range issues such as contradiction, topic drift, and entity inconsistency. To address this, we propose SKG-Eval, a quasi-deterministic and interpretable framework that models dialogue as an evolving Semantic Knowledge Graph (SKG) of entities, relations, and commitments across turns. The framework incrementally updates the graph through structured triple extraction and computes three complementary signals: (i) local relevance, measuring alignment with the current prompt and optional reference; (ii) historical consistency, evaluating how newly introduced information connects to prior conversational context using graph-based and embedding-driven signals; and (iii) logical coherence, assessed by a geometric contradiction engine that detects cross-turn conflicts without relying on NLI models or LLM judges. These signals are adaptively fused and aggregated into a length-invariant session score via recency-weighted trend analysis. Across multiple benchmarks, SKG-Eval achieves higher correlation with human judgments and substantially improves detection of long-range inconsistencies in extended conversations. In addition, the framework produces explicit contradiction certificates and deterministic scores for fixed inputs, enabling reproducible and auditable evaluation. Overall, our results suggest that structured externalized state tracking through semantic knowledge graphs provides a scalable alternative to implicit reasoning in LLM-based dialogue evaluators.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。