现有时间知识图谱评估存在漏洞，新基准可更真实地衡量演化建模能力。
Towards Better Evolution Modeling for Temporal Knowledge Graphs
- 设计新基准纠正数据集偏差，避免仅靠共现计数就能高分的陷阱
- 在YAGO数据集上，新方法使Hits@10提升至0.9以上，但需真正利用时序信息
- 适合研究时间知识图谱演化建模、评估机制改进的学者使用
时间知识图谱（TKG）结构化地保存随时间演化的知识。近期研究致力于设计模型以学习TKG的演化特性，用于预测未来事实，取得显著成果，例如在YAGO数据集上Hits@10分数超过0.9。然而，我们发现现有评估基准无意中引入了捷径：仅通过统计共现即可达到接近顶尖性能，无需使用任何时序信息。本文分析该问题根源，识别出当前数据集中的固有偏差及评估任务过于简化，导致此类捷径可被利用。进一步揭示现有基准的其他缺陷，包括时间区间知识格式不合理、忽略知识过时现象，以及缺乏精确理解演化的充分信息，这些都会加剧捷径效应并阻碍公平评估。为此，我们提出新的TKG演化评估基准，包含四个去偏数据集和两个贴近演化过程的新任务，促进对TKG演化建模挑战的准确理解。基准代码与数据已公开于https://github.com/zjs123/TKG-Benchmark。
原文摘要 · Abstract (English)
Temporal knowledge graphs (TKGs) structurally preserve evolving human knowledge. Recent research has focused on designing models to learn the evolutionary nature of TKGs to predict future facts, achieving impressive results. For instance, Hits@10 scores over 0.9 on YAGO dataset. However, we find that existing benchmarks inadvertently introduce a shortcut. Near state-of-the-art performance can be simply achieved by counting co-occurrences, without using any temporal information. In this work, we examine the root cause of this issue, identifying inherent biases in current datasets and over simplified form of evaluation task that can be exploited by these biases. Through this analysis, we further uncover additional limitations of existing benchmarks, including unreasonable formatting of time-interval knowledge, ignorance of learning knowledge obsolescence, and insufficient information for precise evolution understanding, all of which can amplify the shortcut and hinder a fair assessment. Therefore, we introduce the TKG evolution benchmark. It includes four bias-corrected datasets and two novel tasks closely aligned with the evolution process, promoting a more accurate understanding of the challenges in TKG evolution modeling. Benchmark is available at: https://github.com/zjs123/TKG-Benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。