提出模糊匹配的全析取集成方法，提升异源数据融合效果。
Fuzzy Integration of Data Lake Tables
- 引入模糊匹配机制，扩展全析取连接支持近似值合并。
- 实验表明该方法在主流实现上仅增加少量时间开销。
- 适合处理真实场景中存在同义词、缩写等不一致数据的集成任务。
数据集成是数据科学流程中的关键步骤，旨在统一不同数据集的信息以支持全面分析。全析取（Full Disjunction）作为外连接算子的关联扩展，能有效保留并整合所有可用信息。现有全析取算法仅适用于等值连接场景，即仅当连接字段值完全相同时才进行合并。然而在开放数据环境中，数据来源多样，存在值不一致（如同义词、缩写等）且元数据有限，仅依赖等值匹配严重限制了全析取的融合能力。为此，本文提出全析取的扩展，支持对近似或模糊匹配的记录进行集成。我们设计了一种新型数据驱动方法，使全析取能够处理非精确匹配。实验结果表明，模糊全析取相比先进全析取实现仅增加轻微时间开销，同时显著提升了数据集成的有效性。
原文摘要 · Abstract (English)
Data integration is an important step in any data science pipeline where the objective is to unify the information available in different datasets for comprehensive analysis. Full Disjunction, which is an associative extension of the outer join operator, has been shown to be an effective operator for integrating datasets. It fully preserves and combines the available information. Existing Full Disjunction algorithms only consider the equi-join scenario where only tuples having the same value on joining columns are integrated. This, however, does not realistically represent an open data scenario, where datasets come from diverse sources with inconsistent values (e.g., synonyms, abbreviations, etc.) and with limited metadata. So, joining just on equal values severely limits the ability of Full Disjunction to fully combine datasets. Thus, in this work, we propose an extension of Full Disjunction to also account for "fuzzy" matches among tuples. We present a novel data-driven approach to enable the joining of approximate or fuzzy matches within Full Disjunction. Experimentally, we show that fuzzy Full Disjunction does not add significant time overhead over a state-of-the-art Full Disjunction implementation and also that it enhances the integration effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。