首个对开发环境配置过程进行细粒度评估的基准,揭示Agent失败关键环节。
Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents
- 通过注入真实README错误构建可复现的配置任务
- 发现代理能定位问题但难根据反馈有效修复
- 适合改进LLM Agent在工程自动化中的落地能力
基于大语言模型的智能体在软件工程中展现潜力,但环境配置仍受限于人工投入高、缺乏大规模高质量数据集。现有基准仅评估最终构建/测试是否成功,无法揭示代理在配置过程中的具体表现。本文提出环境配置诊断基准Enconda-bench,支持对代理在规划、感知错误、反馈修复和执行配置等环节的细粒度轨迹分析。任务实例通过注入真实存在的README错误自动生成,并在Docker中验证,确保可扩展性和评估质量。该基准结合过程级分析与端到端可执行性,超越单纯的成功率统计。对主流LLM与代理框架的评估表明,代理虽能定位错误,但难以将反馈转化为有效修复,制约整体性能。据我们所知,Enconda-bench是首个提供环境配置内部能力过程级评估的框架,为改进软件工程智能体提供可操作洞察。
原文摘要 · Abstract (English)
Large language model-based agents show promise for software engineering, but environment configuration remains a bottleneck due to heavy manual effort and scarce large-scale, high-quality datasets. Existing benchmarks assess only end-to-end build/test success, obscuring where and why agents succeed or fail. We introduce the Environment Configuration Diagnosis Benchmark, Enconda-bench, which provides process-level trajectory assessment of fine-grained agent capabilities during environment setup-planning, perception-driven error diagnosis, feedback-driven repair, and action to execute final environment configuration. Our task instances are automatically constructed by injecting realistic README errors and are validated in Docker for scalable, high-quality evaluation. Enconda-bench combines process-level analysis with end-to-end executability to enable capability assessments beyond aggregate success rates. Evaluations across state-of-the-art LLMs and agent frameworks show that while agents can localize errors, they struggle to translate feedback into effective corrections, limiting end-to-end performance. To our knowledge, Enconda-bench is the first framework to provide process-level internal capability assessment for environment configuration, offering actionable insights for improving software engineering agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。