arXiv:2502.13069cs.AI2025-02中稿 · ICLR被引 31

让AI代码助手学会主动问清模糊指令,提升生成质量。

Ambig-SWE: Interactive Agents to Overcome Underspecificity in Software Engineering

  • 设计交互式流程识别并追问模糊代码需求
  • 交互后性能最高提升74%,显著优于不提问
  • 适合关注AI辅助编程可靠性的开发者与研究者

AI代理在自动化任务时常面临用户指令不明确的问题。若盲目假设或不询问澄清,可能导致错误结果、工具误用及资源浪费。本文研究大模型代理在交互式代码生成中处理模糊指令的能力,评估其在三个关键步骤的表现:(a) 检测指令不明确,(b) 提出针对性澄清问题,(c) 通过交互提升表现。提出Ambig-SWE,作为SWE-Bench Verified的模糊版本,专门用于评估代理在歧义与交互下的行为。结果显示,模型难以区分明确与不明确指令;但在交互场景下,能有效获取用户信息,性能相比非交互设置最高提升74%,凸显有效交互的价值。研究揭示当前顶尖模型在复杂软件任务中应对缺失信息的关键短板,并将评估拆分为可优化的独立步骤。

原文摘要 · Abstract (English)

AI agents are increasingly being deployed to automate tasks, often based on underspecified user instructions. Making unwarranted assumptions to compensate for the missing information and failing to ask clarifying questions can lead to suboptimal outcomes, safety risks due to tool misuse, and wasted computational resources. In this work, we study the ability of LLM agents to handle underspecified instructions in interactive code generation settings by evaluating proprietary and open-weight models on their performance across three key steps: (a) detecting underspecificity, (b) asking targeted clarification questions, and (c) leveraging the interaction to improve performance in underspecified scenarios. We introduce Ambig-SWE, an underspecified variant of SWE-Bench Verified, specifically designed to evaluate agent behavior under ambiguity and interaction. Our findings reveal that models struggle to distinguish between well-specified and underspecified instructions. However, when models interact for underspecified inputs, they effectively obtain vital information from the user leading to significant improvements in performance, up to 74% over the non-interactive settings, underscoring the value of effective interaction. Our study highlights critical gaps in how current state-of-the-art models handle missing information in complex software engineering tasks and structures the evaluation into distinct steps to enable targeted improvements.

AI编程交互设计模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。