测试智能体自动搭建科研代码运行环境的能力，填补了真实科研场景的评估空白。
ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
- 给定科研仓库和文档，让智能体自动生成可运行的执行环境
- 当前顶尖智能体在依赖解析和版本兼容上失败率高，暴露关键短板
- 适合研究自动化科研、AI编程助手或环境配置的开发者参考
自主智能体正被期待支持科学研究所需的代码执行，现有基准多聚焦代码修复与自主实验，但通常假设执行环境已预配置。实际科研中需解决复杂的软件依赖、硬件与框架版本对齐及分布式配置等问题，而此能力尚未被充分评估。本文提出ResearchEnvBench，一个面向科研代码执行中环境合成的基准测试。给定研究仓库、文档及目标执行设置，智能体需在运行时构建出可成功执行的环境。对多样化科研仓库的评估显示，当前最先进智能体存在显著差距，主要失败原因包括依赖解析不完整与版本耦合脆弱。ResearchEnvBench为推动智能体实现可复现的科学研究提供了真实可信的测试平台。
原文摘要 · Abstract (English)
Autonomous agents are increasingly expected to support scientific research, and recent benchmarks report progress in code repair and autonomous experimentation. However, these evaluations typically assume a pre-configured execution environment, which requires resolving complex software dependencies, aligning hardware and framework versions, and configuring distributed execution, yet this capability remains largely unbenchmarked. We introduce ResearchEnvBench, a benchmark for environment synthesis in research code execution. Given a research repository, documentation, and a target execution setting, agents must construct an environment that successfully executes at runtime. Evaluations on diverse research repositories reveal a substantial gap in current SOTA agents, with failures dominated by incomplete dependency resolution and brittle version coupling. ResearchEnvBench provides a realistic testbed for advancing autonomous agents toward reproducible scientific research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。