评测大模型用自然语言做数据清洗的能力,发现距离实用还有很大差距。
PrepBench: How Far Are We from Natural-Language-Driven Data Preparation?

- 设计新基准测试自然语言驱动的数据准备任务
- 多数任务需100行以上代码,最长超300行
- 适合关注大模型在数据分析落地的开发者
数据准备是数据分析流程中核心且耗时的环节。传统工具依赖图形界面(GUI)让用户通过可视化操作定义转换流程。近年来大语言模型(LLM)的发展带来了自然语言(NL)驱动数据准备的新可能,用户可直接用自然语言描述需求。但当前大模型在实际应用中距离这一范式转变仍有显著差距。现有代码生成基准未能涵盖数据准备的关键特性,如用户意图模糊、真实数据不完整,以及将代码转化为可解释工作流以供验证的需求。为此,我们提出PrepBench,一个评估自然语言驱动数据准备能力的基准,涵盖三个核心能力:交互式澄清歧义、准备代码生成和代码到工作流的转换。数据源自Preppin' Data Challenges,并经过系统化扩展。基准覆盖多个领域,每项任务包含3至18个数据处理步骤,近半数任务需超过100行Python代码,最长解决方案接近300行。评估结果表明,尽管近期进展显著,当前最先进的大模型仍难以有效实现该范式。PrepBench为衡量这一差距提供了系统性基准,有助于识别实现自然语言驱动数据准备的关键挑战。
原文摘要 · Abstract (English)
Data preparation is a central and time-consuming stage in data analysis workflows. Traditionally, commercial tools have relied on graphical user interfaces (GUIs) to simplify data preparation, allowing users to define transformations through visual operators and workflows. Recent advances in large language models (LLMs) raise the possibility of a paradigm shift toward natural language (NL)-driven data preparation, in which users can specify preparation intents in NL directly. However, it remains unclear how far current LLM-based agents are from this paradigm shift in practice. Existing code generation benchmarks do not capture key characteristics of data preparation, including ambiguous user intents, imperfect real-world data, and the need to translate code into interpretable workflows for validation. To bridge this gap, we present PrepBench, a benchmark designed to evaluate NL-driven data preparation along three core capabilities: interactive disambiguation, prep-code generation, and code-to-workflow translation. We crawl data from the Preppin' Data Challenges, and then extend it into a systematically designed benchmark. The benchmark covers diverse domains, and each task involves 3 to 18 data preparation steps. Nearly half of the tasks require over 100 lines of Python code, and the longest solutions approach 300 lines. Our evaluation shows that, despite recent progress, realizing this paradigm shift remains challenging for state-of-the-art LLMs. PrepBench provides a principled benchmark for measuring this gap and helps identify key challenges toward realizing NL-driven data preparation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。