为大模型驱动的开发助手提供可交互的调试工具,让开发者像看程序一样看清楚它的每一步操作。
AgentStepper: Interactive Debugging of Software Development Agents
- 将代理执行过程转化为可交互的对话式轨迹,支持断点和逐步执行
- 用户研究显示调试效率提升,找错成功率从17%升至60%
- 适合需要调试AI开发助手的工程师和研究人员
由大语言模型(LLM)驱动的软件开发助手在自动化环境配置、问题求解和程序修复方面展现出巨大潜力。然而,由于其复杂动态的特性,理解和调试这些助手仍具挑战性。开发者需分析LLM查询、工具调用和代码修改的完整轨迹,但现有技术难以以清晰直观的方式呈现中间过程。本文提出关键洞察:调试软件开发助手与传统程序调试有诸多相似之处,但需更高层次的抽象,从底层实现细节转向高层代理行为。基于此,我们提出AgentStepper,首个面向基于LLM的软件工程代理的交互式调试器。AgentStepper将代理轨迹表示为LLM、代理程序与工具间的结构化对话,支持断点、单步执行和实时编辑提示词与工具调用,同时捕获并展示代码仓库级别的中间变更。评估在三个前沿开发代理(ExecutionAgent、SWE-Agent、RepairAgent)上进行,集成仅需少量代码修改(39-42行)。此外,对12名参与者的用户研究显示,相比传统工具,AgentStepper显著提升轨迹理解能力(平均性能64% vs. 67%),找错成功率从17%提升至60%,并降低感知工作量(如挫败感从5.4/7.0降至2.4/7.0)。
原文摘要 · Abstract (English)
Software development agents powered by large language models (LLMs) have shown great promise in automating tasks like environment setup, issue solving, and program repair. Unfortunately, understanding and debugging such agents remain challenging due to their complex and dynamic nature. Developers must reason about trajectories of LLM queries, tool calls, and code modifications, but current techniques reveal little of this intermediate process in a comprehensible format. The key insight of this paper is that debugging software development agents shares many similarities with conventional debugging of software programs, yet requires a higher level of abstraction that raises the level from low-level implementation details to high-level agent actions. Drawing on this insight, we introduce AgentStepper, the first interactive debugger for LLM-based software engineering agents. AgentStepper enables developers to inspect, control, and interactively manipulate agent trajectories. AgentStepper represents trajectories as structured conversations among an LLM, the agent program, and tools. It supports breakpoints, stepwise execution, and live editing of prompts and tool invocations, while capturing and displaying intermediate repository-level code changes. Our evaluation applies AgentStepper to three state-of-the-art software development agents, ExecutionAgent, SWE-Agent, and RepairAgent, showing that integrating the approach into existing agents requires minor code changes (39-42 edited lines). Moreover, we report on a user study with twelve participants, indicating that AgentStepper improves the ability of participants to interpret trajectories (64% vs. 67% mean performance) and identify bugs in the agent's implementation (17% vs. 60% success rate), while reducing perceived workload (e.g., frustration reduced from 5.4/7.0 to 2.4/7.0) compared to conventional tools.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。