真实用户请求与现有评测差距大,明确目标可显著提升代码模型表现
RealSWE: A Compositional Evaluation of Coding Agents under Realistic User Requests

- 构建六类信息分类与四维度语言风格体系,分析真实请求与评测数据差异
- 88%真实请求仅含简短问题描述,而评测中仅7%如此;87%真实请求随意,评测中94%正式
- 强调目标和动机能大幅提升模型表现,语言风格影响较小,适合开发者参考
当前代码代理普遍在 SWE-bench 系列基准上评估,其任务源自精心筛选的 GitHub 问题:长、结构化且信息丰富。但真实用户请求通常更短且非结构化。为此,我们提出六类信息分类与四个语言风格维度,并应用于 SWE-chat 中的真实用户提示,以及 SWE-bench Verified 和 Pro 的问题陈述。结果发现,仅包含问题陈述(可能附加有限上下文)的请求占真实提示的 88%,而在基准中仅占 7%;87% 的真实提示为非正式表达,而基准中 94% 为正式。基于此,我们引入 RealSWE,包含从 SWE-bench Verified 和 Pro 派生的 381 个多变体任务族。每个家族共享相同底层任务与正确补丁,仅在信息构成与语言风格上不同。对七种主流 LLM 在 RealSWE 上的评估显示:(i) 真实输入使解决率平均下降 6.4 个百分点,且可能改变模型排名;(ii) 明确包含期望行为与动机显著提升性能,而环境信息与复现步骤仅增加词数,无明显收益;(iii) 语言风格影响较小且依赖模型。研究为用户与代理提供实用建议:明确目标与动机可显著提升软件工程能力。
原文摘要 · Abstract (English)
Coding agents are now commonly evaluated on the SWE-bench family of benchmarks, whose tasks are built from curated GitHub issues: long, structured, and information-rich. Real user requests, however, are typically far shorter and less structured. To characterize this gap, we define a six-category information taxonomy and four dimensions of linguistic style, and apply them to real user prompts from SWE-chat and problem statements from SWE-bench Verified and Pro. We find that requests carrying only a problem statement, alone or with limited additional context, account for 88% of real prompts but just 7% of benchmark problems. Furthermore, 87% of real prompts are casually written whereas 94% of benchmark problems are formal. Guided by these observations, we introduce RealSWE, 381 multi-variant task families derived from SWE-bench Verified and Pro. Variants within each family share the same underlying task and gold patch while differing only in information composition and linguistic style. Evaluating seven contemporary LLMs with RealSWE, we find that i) realistic inputs reduce resolution rates by 6.4 pp on average and can change model rankings. Controlled analysis further shows that ii) including Desired Behavior and Motivation significantly affects performance, whereas Environment Information and Reproduction Steps merely add tokens without measurable benefit; iii) linguistic style has only small, model-dependent effects. These findings provide actionable guidance for users and agents: explicitly stating the desired behavior and motivation, which most real prompts omit, substantially improves the LLM's software engineering performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。