arXiv:2510.25694cs.SEcs.AI2025-10被引 18

首个对开发环境配置过程进行细粒度评估的基准,揭示Agent失败关键环节。

Process-Level Trajectory Evaluation for Environment Configuration in Software Engineering Agents

  • 通过注入真实README错误构建可复现的配置任务
  • 发现代理能定位问题但难根据反馈有效修复
  • 适合改进LLM Agent在工程自动化中的落地能力

基于大语言模型的智能体在软件工程中展现潜力,但环境配置仍受限于人工投入高、缺乏大规模高质量数据集。现有基准仅评估最终构建/测试是否成功,无法揭示代理在配置过程中的具体表现。本文提出环境配置诊断基准Enconda-bench,支持对代理在规划、感知错误、反馈修复和执行配置等环节的细粒度轨迹分析。任务实例通过注入真实存在的README错误自动生成,并在Docker中验证,确保可扩展性和评估质量。该基准结合过程级分析与端到端可执行性,超越单纯的成功率统计。对主流LLM与代理框架的评估表明,代理虽能定位错误,但难以将反馈转化为有效修复,制约整体性能。据我们所知,Enconda-bench是首个提供环境配置内部能力过程级评估的框架,为改进软件工程智能体提供可操作洞察。

原文摘要 · Abstract (English)

Large language model-based agents show promise for software engineering, but environment configuration remains a bottleneck due to heavy manual effort and scarce large-scale, high-quality datasets. Existing benchmarks assess only end-to-end build/test success, obscuring where and why agents succeed or fail. We introduce the Environment Configuration Diagnosis Benchmark, Enconda-bench, which provides process-level trajectory assessment of fine-grained agent capabilities during environment setup-planning, perception-driven error diagnosis, feedback-driven repair, and action to execute final environment configuration. Our task instances are automatically constructed by injecting realistic README errors and are validated in Docker for scalable, high-quality evaluation. Enconda-bench combines process-level analysis with end-to-end executability to enable capability assessments beyond aggregate success rates. Evaluations across state-of-the-art LLMs and agent frameworks show that while agents can localize errors, they struggle to translate feedback into effective corrections, limiting end-to-end performance. To our knowledge, Enconda-bench is the first framework to provide process-level internal capability assessment for environment configuration, offering actionable insights for improving software engineering agents.

软件工程LLM Agent环境配置评估基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。