自动学习表格语义约束，无需专家干预即可发现数据错误。
Auto-Test: Learning Semantic-Domain Constraints for Unsupervised Error Detection in Tables
- 从大规模表格语料中自动推导语义域约束，无需人工标注。
- 在2400个真实数据列上验证，可直接检测实际表格中的错误。
- 适合数据清洗初学者或缺乏领域专家的团队使用。
数据清洗是数据管理中的长期挑战。现有算法虽强大，但通常依赖领域专家手动为每张表定义数据质量约束后才能应用。本文提出一类新型数据质量约束——语义域约束（Semantic-Domain Constraints），可从表格语料库中自动推断并应用于任意表格，无需人工逐表指定。我们构建了一个基于大规模统计检验的系统性框架，通过优化框架提炼出核心约束集，并具备可证明的质量保证。大量实验表明，该约束类可直接用于真实表格中的错误检测，也可作为现有专家驱动清洗技术的互补补充。我们还发布了包含2400个真实数据列的标注基准数据集及代码，网址：https://github.com/qixuchen/AutoTest，以推动后续研究。
原文摘要 · Abstract (English)
Data cleaning is a long-standing challenge in data management. While powerful logic and statistical algorithms have been developed to detect and repair data errors in tables, existing algorithms predominantly rely on domain-experts to first manually specify data-quality constraints specific to a given table, before data cleaning algorithms can be applied. In this work, we propose a new class of data-quality constraints that we call Semantic-Domain Constraints, which can be reliably inferred and automatically applied to any tables, without requiring domain-experts to manually specify on a per-table basis. We develop a principled framework to systematically learn such constraints from table corpora using large-scale statistical tests, which can further be distilled into a core set of constraints using our optimization framework, with provable quality guarantees. Extensive evaluations show that this new class of constraints can be used to both (1) directly detect errors on real tables in the wild, and (2) augment existing expert-driven data-cleaning techniques as a new class of complementary constraints. Our extensively labeled benchmark dataset with 2400 real data columns, as well as our code are available at https://github.com/qixuchen/AutoTest to facilitate future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。