用大模型自动生成数据清洗代码，提升效率与准确性。
Data Wrangling Task Automation Using Code-Generating Language Models
- 基于大模型生成可执行代码，自动完成数据清洗任务。
- 支持缺失值填补、错误检测与修正，适配多种数据场景。
- 无需额外训练，适合科研与工程人员快速处理数据。
大型表格数据集中的数据质量保障是一项关键挑战，通常通过数据清洗任务来解决。传统统计方法虽高效，但难以理解语义上下文；深度学习方法则资源消耗大，需针对特定任务和数据集进行训练。为克服这些局限，我们提出一种自动化系统，利用大语言模型生成可执行代码，完成缺失值填补、错误检测与修正等任务。该系统旨在识别数据内在模式，并结合外部知识，有效应对依赖记忆与不依赖记忆的任务。
原文摘要 · Abstract (English)
Ensuring data quality in large tabular datasets is a critical challenge, typically addressed through data wrangling tasks. Traditional statistical methods, though efficient, cannot often understand the semantic context and deep learning approaches are resource-intensive, requiring task and dataset-specific training. To overcome these shortcomings, we present an automated system that utilizes large language models to generate executable code for tasks like missing value imputation, error detection, and error correction. Our system aims to identify inherent patterns in the data while leveraging external knowledge, effectively addressing both memory-dependent and memory-independent tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。