首个在真实电脑环境评估数据科学全流程自动化的基准,揭示当前智能体能力严重不足。
DSAgentBench: Can Agents Automate End-to-End Data-Science Workflows in Real Computer Environments?

- 构建真实计算机环境中的全流程数据科学任务集,需跨工具协同与多步推理
- 最强模型仅达56.7%成功率,开源模型普遍低于1%
- 适合研究具身智能、自动化数据分析与智能体评估的学者
真实世界的数据科学涉及从数据清洗、探索、建模、可视化到验证的长周期流程,需在真实操作系统中协调使用笔记本、IDE、终端、浏览器和数据库等工具。现有基准缺乏真实计算机交互,无法评估智能体是否能在现实环境中执行完整的端到端数据科学工作流,难以反映实际工作中多阶段、多工具协作的复杂性。我们提出DSAgentBench,首个在真实计算环境中评估智能体自动化完整数据科学流程的基准。该基准包含275个多样化任务,覆盖数据科学全生命周期,每项任务要求基于中间输出进行决策并协调多工具操作,并配备确定性评估器,验证分析正确性、视觉输出及模型性能,而非仅代码执行。对15个闭源与开源模型的广泛实验表明,即使最强模型Claude-4.6-Sonnet也仅达56.70%任务成功率,所有开源模型均低于1%,主要失败于工具编排、操作系统上下文理解与多步推理。结果揭示了当前智能体系统与真实数据科学流程之间存在显著能力差距,确立了DSAgentBench作为发展可落地、可验证、自主数据科学智能体的基础。项目已开源:https://github.com/vis-nlp/DSAgentBench。
原文摘要 · Abstract (English)
Real-world data science involves long-horizon workflows that span data wrangling, exploration, modeling, visualization, and validation, and require coordinated use of tools such as notebooks, IDEs, terminals, browsers, and databases within real operating environments. Yet existing benchmarks lack real-computer interaction and do not evaluate whether agents can execute complete end-to-end data-science workflows in realistic computing environments, failing to capture the multi-stage, multi-tool nature of data-science practice. We introduce DSAgentBench, the first benchmark to evaluate whether agents can automate full data-science workflows inside real computer environments. DSAgentBench contains 275 diverse tasks covering the entire data-science life-cycle, reflecting the complexity and tool coordination required in practice. Each task requires grounding decisions in intermediate outputs and coordinated tool use, and includes a deterministic evaluator that verifies analytical correctness, visual outputs, and model performance rather than code-only execution. Our extensive experiments with 15 closed- and open-source models show that even the strongest agent, Claude-4.6-Sonnet, achieves only 56.70% task success, while all open-source agents remain below 1%, frequently failing at tool orchestration, OS grounding, and multi-step reasoning. These results reveal a substantial capability gap between current agentic systems and real data-science workflows, positioning DSAgentBench as a foundation for developing grounded, verifiable, autonomous data-science agents. We release DSAgentBench at https://github.com/vis-nlp/DSAgentBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。