提出标准化用户协议，让知识图谱构建研究可比可复现。
A Protocol for KG Construction Tasks Involving Users
- 设计统一任务、参与者与评估指标的用户研究协议
- 支持RML核心功能的跨语言工具对比，提升可比性
- 为后续扩展比较提供可复用框架，适合评测类研究者
从（半）结构化数据构建知识图谱（KGC）具有挑战性，而让用户参与是该领域常被提及的问题。尽管我们在知识图谱构建语言和工具方面取得进展，但不同研究采用的协议差异巨大，导致难以比较各类语言、技术与工具。本文分析现有用户研究，发现任务设计、参与者选择和评估指标缺乏系统一致性，且数据分析与结果报告方法也未标准化。为此，我们提出一套用户协议，旨在解决上述问题。协议参考文献中合适元素，聚焦RML核心功能，覆盖多数前沿技术与工具。同时提出如何扩展协议以比较RML扩展功能。该协议为知识图谱构建用户研究的可比性评估迈出关键一步。
原文摘要 · Abstract (English)
Knowledge graph construction (KGC) from (semi-)structured data is challenging, and facilitating user involvement is an issue frequently brought up within this community. We cannot deny the progress we have made with respect to (declarative) knowledge graph construction languages and tools to help build such mappings. However, it is surprising that no two studies report on similar protocols. This heterogeneity does not allow for comparing KGC languages, techniques, and tools. This paper first analyses studies involving users to identify the points of comparison. These gaps include a lack of systematic consistency in task design, participant selection, and evaluation metrics. Moreover, there needs to be a systematic way of analyzing the data and reporting the findings, which is also lacking. We thus propose and introduce a user protocol for KGC designed to address this challenge. Where possible, we draw and take elements from the literature we deem fit for such a protocol. The protocol, as such, allows for the comparison of languages and techniques for the RDF Mapping Language (RML) core functionality, which is covered by most of the other state-of-the-art techniques and tools. We also propose how the protocol can be amended to compare extensions (of RML). This protocol provides an important step towards a more comparable evaluation of KGC user studies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。