首个面向真实工业数据工程的智能体评估基准，验证自主处理能力。
DataClawEval: A Benchmark for Data Engineering Agents in Real Industrial Harness
- 基于真实企业代码构建100个端到端任务，覆盖5种主流引擎。
- 最强模型仅达74.9分，无通用优胜者，体现严格领域依赖性。
- 适合研究自动化数据流水线、智能体评估的开发者与研究者。
大语言模型（LLM）及其代理正被广泛用于自动化复杂工作流，有望重塑数据管理与处理。然而，现有基准多聚焦于简化版文本转SQL或数据分析任务，对端到端数据工程这一关键且复杂的领域关注不足。为此，我们提出DataClawEval，首个专为评估自主代理在真实数据工程场景中端到端任务完成能力而设计的综合性基准。该基准基于专业企业数据工程师编写的生产级代码，包含100个严谨的端到端任务，覆盖PySpark、MySQL、HiveSQL、PrestoSQL/Trino和FlinkSQL五种执行引擎。每个任务在专属隔离沙箱中运行，由确定性规则脚本评分，而非非确定性的LLM评判。对16个前沿代理的评估揭示了显著局限：最强模型整体得分仅74.9，且无单一模型在所有引擎上领先，各模型仅在特定引擎表现突出，表明当前自主数据工程仍面临严峻挑战。我们已将数据集、容器化环境及确定性评估脚本开源至https://github.com/Dicemy/DataClawEval/tree/master。
原文摘要 · Abstract (English)
Large language models (LLMs) and LLM-based agents are increasingly being deployed to automate complex workflows, promising to revolutionize data management and processing. However, existing benchmarks predominantly focus on simplified Text-to-SQL translation or data analysis, leaving the critical and complex domain of end-to-end data engineering largely unexplored. To bridge this gap, we introduce DataClawEval, the first comprehensive benchmark designed specifically to evaluate the end-to-end task completion capabilities of autonomous agents in real-world data engineering scenarios. Built upon production-grade code authored by professional enterprise data engineers, it comprises 100 rigorous, end-to-end tasks spanning five execution engines: PySpark, MySQL, HiveSQL, PrestoSQL/Trino, and FlinkSQL. Rather than non-deterministic LLM-as-a-judge scoring, each task is executed within a case-specific, isolated sandbox and graded by deterministic, rule-based scripts. Evaluating 16 frontier agents exposes critical limitations: The strongest model attains only 74.9 overall, and no single model dominates, as each excels on a different engine, revealing strict domain specialization rather than omnipotent proficiency. Thus, autonomous data engineering remains a formidable, unresolved challenge. We release our dataset, containerized environments, and deterministic evaluation scripts at https://github.com/Dicemy/DataClawEval/tree/master
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。