评测大模型在真实软件工程任务中作为开发助手的表现。
IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks
- 构建可运行的IDE环境，模拟真实开发工具链。
- 在80个未公开项目中测试多语言全栈任务，准确率超基准。
- 首次实现意图与代码修改的系统性关联分析，适合研究者和开发者参考。
IDE-Bench 是一个用于评估大语言模型作为 IDE 代理在真实软件工程任务上表现的综合框架。通过 Docker 化测试环境，提供代码库搜索、结构化文件编辑及全栈应用测试等高阶抽象，使模型能以类原生 IDE 方式协作。我们构建了涵盖 C/C++、Java、MERN 技术栈的 80 个未发布项目任务，覆盖功能开发、缺陷修复、重构与性能优化等日常开发场景。该基准是首个在多语言、全栈环境下，对完全未污染代码进行系统性意图-修改关联分析的评测体系。相关数据与公开排行榜已发布于 https://ide-bench.com。
原文摘要 · Abstract (English)
IDE-Bench is a comprehensive framework for evaluating AI IDE agents on real-world software engineering tasks through an IDE-native tool interface. We present a Dockerized test harness that goes beyond raw terminal execution, granting models a structured tool ecosystem that represents AI-native IDEs like Cursor and Windsurf. By providing high-level abstractions for codebase search, structured file editing, and tools for testing full-stack applications, IDE-Bench evaluates an agent's ability to act as a true engineering collaborator. For evaluation and to prevent training data contamination, we created 80 tasks across eight never-published repositories spanning C/C++, Java, and MERN stacks, representing modern tech stack production scenarios, including feature implementation, bug fixing, refactoring, and performance optimization that mirror daily developer workflows in private codebases. Our benchmark is the first to systematically correlate agent-reported intent with successful project-level modifications in a multi-language, full-stack environment on completely uncontaminated code. We release IDE-Bench and a public leaderboard at: https://ide-bench.com.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。