arXiv:2609.06053cs.LG2026-09

用大模型自动生成数据质量校验规则,减少人工遗漏。

Data Quality Rule Generation with LLMs

论文配图:Data Quality Rule Generation with LLMs
图 1 · 摘自论文原文
  • 大模型生成候选规则,再通过四类过滤确保可用性
  • 在多个数据集上生成了有效且精简的规则集
  • 适合需要高效维护数据质量规则的企业用户

数据验证在企业中至关重要,错误数据可能导致严重后果。现有基于规则的工具虽高效可解释,但手动维护规则集困难,尤其在复杂场景和大数据量下易遗漏关键规则。本文提出LeDQeR,一种基于大语言模型的自动化数据质量规则生成方法。首先,LLM根据脏数据样本生成符合规则语法的候选规则;其次,采用四种过滤技术确保规则的可执行性、正确性、泛化能力并避免冗余。大量实验表明,LeDQeR能在多种数据集和错误类型下生成有效且紧凑的规则集,显著降低人工维护负担。

原文摘要 · Abstract (English)

The validation of data, such as customer and employee data, is an important task in many organizations. Errors in data can have severe consequences. For example, a wrong drug unit in a patient record can lead to life-threatening medication errors, and a missing street number in an address to failed deliveries. Companies often employ rule-based enterprise data quality (DQ) tools, which allow domain experts to specify rules to validate the data over time. While rule-based DQ tools are computationally efficient and provide explainable reports, maintaining a comprehensive rule set manually is challenging, as domain experts often overlook essential rules, especially in complex domains and large data volumes. Hence, closing these gaps remains an open problem in practice. In this paper, we address the challenge of automated DQ rule generation. For this, we formalize a generalizable generate-filter framework and introduce LeDQeR, an LLM-based DQ rule generation approach. First, a large language model (LLM) generates candidates rules from an observed dirty data tuple for a given rule-based DQ tool syntax. Second, we apply four filter techniques that ensure the (i) executability, (ii) correctness, and (iii) generalizability, and avoid (iv) redundancy of the generated rules. An extensive experimental evaluation suggests that LeDQeR is able to produce effective and compact rule sets for various datasets and error types.

数据质量大模型规则生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。