仅用元数据自动推断数据表间关联关系,提升企业数据集成效率。
Nexus: Inferring Join Graphs from Metadata Alone via Iterative Low-Rank Matrix Completion
- 基于低秩矩阵补全与迭代优化,仅凭表名、字段等元数据推断表间连接
- 在四个真实数据集上准确率显著超越现有方法,最快可提速6倍
- 适合缺乏数据值访问权限的企业级数据治理与智能查询系统
自动推断表间连接关系对数据发现、整合、查询和重用至关重要。然而,在大型复杂模式中,尤其在企业环境中无法访问实际数据值时,准确高效地识别这些关系仍具挑战。本文研究仅使用元数据进行连接图推断的问题。通过对大量真实世界模式的实证分析,我们发现以邻接矩阵表示的连接图具有高稀疏性和低秩结构两大特性。基于此,我们将连接图推断建模为低秩矩阵补全问题,并提出Nexus——一种仅依赖元数据的端到端解决方案。为进一步提升精度,我们设计了一种新型期望最大化算法,交替执行低秩矩阵补全与利用大语言模型优化连接候选概率。大量实验表明,Nexus在包括一个真实生产数据集在内的四个数据集上显著优于现有方法。此外,Nexus支持快速模式,结果接近且速度最高达6倍提升,适用于实际部署。
原文摘要 · Abstract (English)
Automatically inferring join relationships is a critical task for effective data discovery, integration, querying and reuse. However, accurately and efficiently identifying these relationships in large and complex schemas can be challenging, especially in enterprise settings where access to data values is constrained. In this paper, we introduce the problem of join graph inference when only metadata is available. We conduct an empirical study on a large number of real-world schemas and observe that join graphs when represented as adjacency matrices exhibit two key properties: high sparsity and low-rank structure. Based on these novel observations, we formulate join graph inference as a low-rank matrix completion problem and propose Nexus, an end-to-end solution using only metadata. To further enhance accuracy, we propose a novel Expectation-Maximization algorithm that alternates between low-rank matrix completion and refining join candidate probabilities by leveraging Large Language Models. Our extensive experiments demonstrate that Nexus outperforms existing methods by a significant margin on four datasets including a real-world production dataset. Additionally, Nexus can operate in a fast mode, providing comparable results with up to 6x speedup, offering a practical and efficient solution for real-world deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。