首个联合评估代码与数据智能的基准，测试智能体处理海量数据任务能力
CODA-BENCH: Can Code Agents Handle Data-Intensive Tasks?
- 构建基于Kaggle生态的海量数据沙箱环境，模拟真实开发场景
- 顶尖智能体在复杂数据任务中成功率仅61.1%，暴露整合能力短板
- 适合研究智能体、自动化编程与数据工程的学者与开发者
随着智能体展现成为自主工程师的潜力，对能反映真实开发复杂性的评估基准的需求日益增长。现实开发环境通常兼具复杂代码与大规模数据（如文件系统），但现有基准多孤立评估代码或数据能力，未能覆盖真实场景。本文提出CODA-BENCH，首个在数据密集型环境中联合评估代码与数据智能的基准。基于Kaggle生态构建的数据密集型Linux沙箱包含数百个数据集，智能体需主动探索复杂文件层级以定位资源，并生成数据驱动分析代码。CODA-BENCH包含1,009个任务，覆盖31个社区，每个任务环境平均含980个文件，模拟真实数据规模与噪声。对先进智能体的评估显示，即使顶尖系统也难以有效融合数据发现与代码执行，成功率为61.1%。结果揭示当前智能体在数据密集型任务中存在显著能力差距，为未来研究指明方向。
原文摘要 · Abstract (English)
Advanced agents are increasingly demonstrating the potential to operate as autonomous engineers, creating a growing demand for evaluation benchmarks that capture the complexity of real-world development. Such environments typically involve both complex code and large-scale data (i.e., file system). However, existing benchmarks usually evaluate code-centric or data-centric capabilities in isolation, leaving a clear gap with real development scenarios. In this paper, we bridge this gap by introducing CODA-BENCH, the first benchmark to jointly evaluate code and data intelligence in a data-intensive environment. We construct a data-intensive Linux sandbox based on the Kaggle ecosystem (containing hundreds of datasets), where agents must actively explore complex file hierarchies to identify relevant resources and generate code for data-driven analytical tasks. CODA-BENCH comprises 1,009 tasks spanning 31 communities, with each task environment containing an average of 980 files, simulating realistic data scale and noise. Evaluations of advanced agents reveal that even top-performing systems struggle to effectively integrate data discovery with code execution, achieving a success rate of only 61.1%. These results highlight a substantial gap in current agentic capabilities for data-intensive tasks and point to promising directions for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。