用知识图谱嵌入自动评估大数据质量,更懂上下文。
Automated Big Data Quality Assessment using Knowledge Graph Embeddings

- 用知识图谱嵌入预测数据与质量规则的关联缺失
- 基于真实传感器数据集验证,生成完整评估方案
- 适合需要精准上下文感知的数据治理场景
自动化数据质量评估对管理大数据至关重要,但现有方法难以实现准确的上下文感知。本文提出一种基于知识图谱的新方法,利用知识图谱嵌入预测输入数据上下文表示与知识图谱中质量规则及维度之间的缺失边。该知识图谱融合了多样化的上下文表征,源自对文献的全面调研,可为不同上下文定制化生成全面的数据质量评估计划。通过注入数值边属性,为每项预测的质量度量分配权重,形成综合评估方案。我们采用AccentureLabs开发并基准测试的AmpliGraph框架进行评估,使用黎巴嫩原子能委员会(LAEC-CNRS)提供的真实辐射传感器数据集。实验结果表明,本方法能够为给定数据集生成全面的数据质量评估方案。
原文摘要 · Abstract (English)
Automated data quality assessment is crucial for managing big data, but existing solutions face challenges in achieving accurate context-aware assessment. This paper presents a novel knowledge-based approach to enhance automated data quality assessment. Our approach utilizes knowledge graph embeddings to predict missing edges between the input dataset's context representation and the relevant quality rules and dimensions within a knowledge graph representing contextual data characteristics and the required quality assessment operations. We surpass conventional practices by integrating diverse representations within the knowledge graph, drawing insights from contextual information from a thorough literature investigation. This integration allows us to develop a comprehensive and context-specific data quality assessment plan tailored to each context. Leveraging the knowledge graph improves our understanding of the input dataset's context, overcoming the limitations of traditional methods that rely solely on strict matching and overlook contextual characteristics. By injecting numerical edge attributes, we assign corresponding weights to each predicted quality measurement, providing a comprehensive data quality assessment plan for the input dataset. To evaluate our approach, we leverage AmpliGraph, a framework developed and benchmarked by AccentureLabs. The evaluation involves employing a real-world radiation sensors dataset provided by the Lebanese Atomic Energy Commission (LAEC-CNRS). The results obtained from this evaluation demonstrate the capability of our solution to generate a comprehensive data quality assessment plan for the given input dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。