构建多语言可验证开发环境，提升大模型代码能力评测可靠性
MEnvAgent: Scalable Polyglot Environment Construction for Verifiable Software Engineering
- 用多智能体架构自动搭建跨语言可执行环境，失败时自主修复
- 在1000个任务上使通过率提升8.6%，耗时减少43%
- 生成了当前最大开源多语言代码测试数据集，适合模型评测
大型语言模型在软件工程领域的应用受限于可验证数据集的稀缺性，根源在于跨语言可执行环境构建复杂。为此，我们提出MEnvAgent——一种支持多语言的自动化环境构建框架，可规模化生成可验证的任务实例。该框架采用规划-执行-验证的多智能体架构，能自主处理构建失败，并引入新型环境复用机制，通过增量修补历史环境降低计算开销。在包含10种语言1000个任务的新基准MEnvBench上的评估显示，MEnvAgent相比基线方法将失败转成功率（F2P）提升8.6%，同时节省43%时间成本。此外，我们基于此框架构建了MEnvData-SWE——目前最大的开源多语言可验证Docker环境数据集，附带解题轨迹，可支持多种模型在广泛SWE任务中实现稳定性能提升。代码、基准和数据集已开源。
原文摘要 · Abstract (English)
The evolution of Large Language Model (LLM) agents for software engineering (SWE) is constrained by the scarcity of verifiable datasets, a bottleneck stemming from the complexity of constructing executable environments across diverse languages. To address this, we introduce MEnvAgent, a Multi-language framework for automated Environment construction that facilitates scalable generation of verifiable task instances. MEnvAgent employs a multi-agent Planning-Execution-Verification architecture to autonomously resolve construction failures and integrates a novel Environment Reuse Mechanism that reduces computational overhead by incrementally patching historical environments. Evaluations on MEnvBench, a new benchmark comprising 1,000 tasks across 10 languages, demonstrate that MEnvAgent outperforms baselines, improving Fail-to-Pass (F2P) rates by 8.6% while reducing time costs by 43%. Additionally, we demonstrate the utility of MEnvAgent by constructing MEnvData-SWE, the largest open-source polyglot dataset of realistic verifiable Docker environments to date, alongside solution trajectories that enable consistent performance gains on SWE tasks across a wide range of models. Our code, benchmark, and dataset are available at https://github.com/ernie-research/MEnvAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。