用真实用户对话重构代码助手评测基准，更准确评估其实际能力
Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
- 通过分析开发者与聊天式编程助手的交互数据，将正式任务转为自然对话查询
- 在多个基准上测试发现，原有评测使模型性能高估超50%（公开）和10-16%（私有）
- 为交互式编程助手评估提供可复用的新范式，适合研究人机协作与评测方法者
当前用于评估软件工程智能体的基准（如 SWE-Bench Verified）主要源自 GitHub 问题，无法真实反映开发者在集成开发环境（IDE）中使用基于聊天的编程助手的实际交互方式。我们提出，这种不匹配会导致对智能体能力的系统性高估，尤其是在修复缺陷方面。为此，我们引入一种新型基准构建框架，通过系统分析开发者与聊天式智能体的交互模式，将现有正式基准转化为更贴近真实用户查询的形式。该方法灵活可扩展，已应用于 SWE-Bench Verified、Multi-SWE-Bench 的 TypeScript 子集及内部私有基准 SWE-Bench C#。基于对主流聊天式智能体交互日志的遥测分析，我们将原始的正式问题描述转换为符合真实用户表达习惯的查询。结果表明，现有基准对部分模型的能力估计过高：在公开基准上超过基线性能 50%，在内部基准上达 10%-16%。本工作确立了通过基准变异技术评估交互式聊天式软件工程智能体的新范式。
原文摘要 · Abstract (English)
Current benchmarks for evaluating software engineering agents, such as SWE-Bench Verified, are predominantly derived from GitHub issues and fail to accurately reflect how developers interact with chat-based coding assistants in integrated development environments (IDEs). We posit that this mismatch leads to a systematic overestimation of agent's capabilities in real-world scenarios, especially bug fixing. We introduce a novel benchmarking framework that transforms existing formal benchmarks into realistic user queries through systematic analysis of developer interaction patterns with chat-based agents. Our methodology is flexible and can be easily extended to existing benchmarks. In this paper, we apply our testing framework to SWE-Bench Verified, the TypeScript subset of Multi-SWE-Bench and a private benchmark, SWE-Bench C# and transform formal GitHub issue descriptions into realistic user-style queries based on telemetry analysis of a popular chat-based agent interactions. Our findings reveal that existing benchmarks significantly overestimate agent capabilities for some models by >50% over baseline performance for public benchmarks and ~10-16% for our internal benchmark. This work establishes a new paradigm for evaluating interactive chat-based software engineering agents through benchmark mutation techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。