用大模型自动生成表格数据质量规则和验证代码，提升自动化水平。
Quality Assessment of Tabular Data using Large Language Models and Code Generation
- 结合聚类与大模型生成语义有效的质量规则
- 通过检索增强生成提升规则准确性，支持可执行代码合成
- 适合需要自动化数据清洗的科研与工程人员
可靠的表格数据质量对下游分析至关重要，但传统基于规则的验证常因效率低、依赖人工干预和计算成本高而受限。本文提出一种三阶段框架，先通过传统聚类过滤数据样本，再迭代式地利用大语言模型（LLM）生成语义合理的质量规则，并由代码生成型大模型合成可执行验证器。为提升规则可靠性，采用检索增强生成（RAG）技术，引入外部知识源与领域特定的少样本示例。通过严格约束机制保障规则与代码片段的一致性与准确性。在基准数据集上的广泛评估验证了该方法的有效性。
原文摘要 · Abstract (English)
Reliable data quality is crucial for downstream analysis of tabular datasets, yet rule-based validation often struggles with inefficiency, human intervention, and high computational costs. We present a three-stage framework that combines statistical inliner detection with LLM-driven rule and code generation. After filtering data samples through traditional clustering, we iteratively prompt LLMs to produce semantically valid quality rules and synthesize their executable validators through code-generating LLMs. To generate reliable quality rules, we aid LLMs with retrieval-augmented generation (RAG) by leveraging external knowledge sources and domain-specific few-shot examples. Robust guardrails ensure the accuracy and consistency of both rules and code snippets. Extensive evaluations on benchmark datasets confirm the effectiveness of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。