首个评估AI生成端到端数据管道能力的基准，揭示当前模型仍严重不足。
ELT-Bench: An End-to-End Benchmark for Evaluating AI Agents on ELT Pipelines
- 构建100个真实场景的完整数据管道任务，涵盖835张源表和203个数据模型。
- 顶尖模型仅3.9%正确生成数据模型，平均耗时89.3步、成本4.3美元。
- 适合研究数据工程自动化、AI Agent能力评估的研究者使用。
随着云数据仓库的普及，提取-加载-转换（ELT）管道在实践中日益重要，但其设计常需大量人工干预以确保正确性。近年来，基于AI的方法在文本转SQL等数据任务中展现出强大能力，为减轻开发负担提供了可能。然而，现有数据工程基准仅评估孤立任务，缺乏对生成完整ELT管道的AI代理的综合评估。为此，我们提出ELT-Bench，一个端到端基准，用于评估AI代理构建完整ELT管道的能力。该基准包含100个管道，覆盖835张源表和203个数据模型，涉及多个领域。通过模拟多样化数据源集成与主流数据工具使用的真实场景，评估代理在数据库交互、代码与SQL编写及流程编排等方面的综合能力。我们采用Spider-Agent和SWE-Agent两个代表性代码代理框架，结合六种主流大语言模型（LLMs）进行测试。结果显示，表现最佳的代理（Spider-Agent + Claude-3.7-Sonnet，启用扩展思维）仅能正确生成3.9%的数据模型，平均每条管道需89.3步操作，成本达4.30美元。实验结果凸显了该基准的挑战性，也表明亟需更先进的AI代理来降低ELT工作流中的手动投入。代码与数据已开源：https://github.com/uiuc-kang-lab/ELT-Bench。
原文摘要 · Abstract (English)
Practitioners are increasingly turning to Extract-Load-Transform (ELT) pipelines with the widespread adoption of cloud data warehouses. However, designing these pipelines often involves significant manual work to ensure correctness. Recent advances in AI-based methods, which have shown strong capabilities in data tasks, such as text-to-SQL, present an opportunity to alleviate manual efforts in developing ELT pipelines. Unfortunately, current benchmarks in data engineering only evaluate isolated tasks, such as using data tools and writing data transformation queries, leaving a significant gap in evaluating AI agents for generating end-to-end ELT pipelines. To fill this gap, we introduce ELT-Bench, an end-to-end benchmark designed to assess the capabilities of AI agents to build ELT pipelines. ELT-Bench consists of 100 pipelines, including 835 source tables and 203 data models across various domains. By simulating realistic scenarios involving the integration of diverse data sources and the use of popular data tools, ELT-Bench evaluates AI agents' abilities in handling complex data engineering workflows. AI agents must interact with databases and data tools, write code and SQL queries, and orchestrate every pipeline stage. We evaluate two representative code agent frameworks, Spider-Agent and SWE-Agent, using six popular Large Language Models (LLMs) on ELT-Bench. The highest-performing agent, Spider-Agent Claude-3.7-Sonnet with extended thinking, correctly generates only 3.9% of data models, with an average cost of $4.30 and 89.3 steps per pipeline. Our experimental results demonstrate the challenges of ELT-Bench and highlight the need for a more advanced AI agent to reduce manual effort in ELT workflows. Our code and data are available at https://github.com/uiuc-kang-lab/ELT-Bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。