arXiv:2508.15357cs.CLcs.PF2025-08被引 2

提出统一评估知识图谱补全模型的新方法,解决多数据集多指标下排名冲突问题。

KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models

  • 设计基于平均解距离的元评估框架,融合多数据集多指标表现
  • 在FB15k-237和WN18RR上验证,生成0到1间可比的统一得分
  • 帮助研究人员公平比较模型,避免因指标差异导致误判

知识图谱(KGs)广泛应用于语义搜索、推荐系统和自然语言处理等领域。由于知识图谱常存在缺失实体与关系的问题,知识图谱补全(KGC)方法被用来预测缺失项。常用的评估指标包括均倒数排名(MRR)、平均秩(MR)和命中率@k(Hit@k)。然而,跨多个数据集和指标比较模型性能面临挑战:某模型在某一数据集表现优异,但在另一数据集表现较差;同一数据集内,不同指标如MRR与Hit@1可能产生矛盾排名,使下游任务模型选择困难。这种不一致性阻碍了全面评估,亟需一种能整合多指标、多数据集表现的统一元评估框架。为此,我们提出基于平均解距离的知识图谱评估方法(KG-EDAS),将模型在多个数据集和多种评估标准下的表现合成一个归一化分数($M_i \in [0,1]$)。相比传统指标仅关注局部表现,EDAS提供全局视角,支持更明智的模型选择,并促进跨数据集评估的公平性。在FB15k-237和WN18RR等基准数据集上的实验表明,EDAS能有效将多指标、多数据集表现整合为一致且稳健的统一排名,构建了一个可靠、可解释、通用性强的KGC模型评估框架。

原文摘要 · Abstract (English)

Knowledge Graphs (KGs) enable applications in various domains such as semantic search, recommendation systems, and natural language processing. KGs are often incomplete, missing entities and relations, an issue addressed by Knowledge Graph Completion (KGC) methods that predict missing elements. Different evaluation metrics, such as Mean Reciprocal Rank (MRR), Mean Rank (MR), and Hit@k, are commonly used to assess the performance of such KGC models. A major challenge in evaluating KGC models, however, lies in comparing their performance across multiple datasets and metrics. A model may outperform others on one dataset but underperform on another, making it difficult to determine overall superiority. Moreover, even within a single dataset, different metrics such as MRR and Hit@1 can yield conflicting rankings, where one model excels in MRR while another performs better in Hit@1, further complicating model selection for downstream tasks. These inconsistencies hinder holistic comparisons and highlight the need for a unified meta-metric that integrates performance across all metrics and datasets to enable a more reliable and interpretable evaluation framework. To address this need, we propose KG Evaluation based on Distance from Average Solution (EDAS), a robust and interpretable meta-metric that synthesizes model performance across multiple datasets and diverse evaluation criteria into a single normalized score ($M_i \in [0,1]$). Unlike traditional metrics that focus on isolated aspects of performance, EDAS offers a global perspective that supports more informed model selection and promotes fairness in cross-dataset evaluation. Experimental results on benchmark datasets such as FB15k-237 and WN18RR demonstrate that EDAS effectively integrates multi-metric, multi-dataset performance into a unified ranking, offering a consistent, robust, and generalizable framework for evaluating KGC models.

知识图谱评估方法多指标融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。