新基准测试让代码模型在用户互动中逐步完成复杂任务，更贴近真实开发。
SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions
- 用模拟用户渐进式给出需求，测试模型在多轮交互中理解意图的能力。
- 顶尖模型在单轮任务上解决率约50%，在多轮互动中仅达25%。
- 适合关注真实开发场景、模型持续学习与纠错能力的研究者。
我们提出SWE-Interact，一个用于评估代码代理在多轮、交互式、用户驱动的软件工程任务中的新测试平台。现有前沿SWE基准通常一次性提供完整需求，评估模型的自主实现能力。而SWE-Interact将模型置于真实的开发者工作流中：由精心设计的用户模拟器从模糊或不完整的指令开始，逐步揭示需求，检查代理的工作空间，并提供针对性反馈、修改意见和新约束，直至任务目标完整传递。该设置基于大规模真实代码代理交互研究，检验模型是否能发现用户意图、适应需求演变并复用已有工作。在一系列前沿及开源权重模型上测试发现，单轮任务表现优异的模型在多轮、用户驱动的任务中无法可靠迁移：最佳模型在单轮基准上解决约50%任务，但在对应SWE-Interact任务中仅解决25%。最强模型（如Opus 4.8和GPT 5.5）虽能在模糊初始指令下良好开局，持续推进直到需求全部浮现，更好地整合信息并写出整洁代码，但仍存在过度主动编码、遗忘需求和技术错误。较弱模型则在模糊环境下开局差，过早放弃，忽略或遗忘指令，反复重写代码。总体而言，SWE-Interact衡量了前沿模型发展中的另一关键维度：用户在环的交互式目标发现与迭代优化能力。
原文摘要 · Abstract (English)
We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。