arXiv:2511.00197cs.SEcs.AI2025-11被引 26

通过分析代码智能体的执行轨迹,揭示其成功与失败的内在规律。

Understanding Code Agent Behaviour: An Empirical Study of Success and Failure Trajectories

  • 对比三种代码智能体的决策路径,发现防御性编程和信息收集策略影响成败。
  • 失败轨迹普遍更长且波动更大,不同智能体的失败模式差异显著。
  • 多数轨迹能准确定位问题文件,但成功关键在于近似修改而非精确修复。

大型语言模型(LLM)智能体在复杂软件工程任务中的部署日益广泛,亟需超越简单成功率的视角来理解其问题求解行为。尽管这些智能体在自动修复问题上表现优异,其决策过程仍不透明。本文对三个前沿代码智能体(OpenHands、SWE-agent、Prometheus)在SWE-Bench基准上的执行轨迹进行实证研究,分析成功与失败案例。研究发现:首先,不同问题解决策略(如防御性编程、上下文收集)在不同场景下决定成败;其次,失败轨迹普遍更长且方差更高,且各智能体失败模式差异明显;第三,故障定位分析显示,即使在失败案例中,72%-81%的轨迹也能正确识别问题文件,但成功更依赖于近似而非精确的代码修改。本研究为通过轨迹分析理解智能体行为提供了基础,助力构建更鲁棒、可解释的自主软件工程系统。

原文摘要 · Abstract (English)

The increasing deployment of Large Language Model (LLM) agents for complex software engineering tasks has created a need to understand their problem-solving behaviours beyond simple success metrics. While these agents demonstrate impressive capabilities in automated issue resolution, their decision-making processes remain largely opaque. This paper presents an empirical study of agent trajectories, namely the execution traces capturing the steps agents take when attempting to resolve software issues. We analyse trajectories from three state-of-the-art code agents (OpenHands, SWE-agent, and Prometheus) on the SWE-Bench benchmark, examining both successful and failed attempts. Our investigation reveals several key insights into agent behaviour. First, we identify how distinct problem-solving strategies, such as defensive programming and context gathering, enable success in different scenarios. Second, we find that failed trajectories are consistently longer and exhibit higher variance than successful ones, with failure patterns differing significantly between agents. Third, our fault localisation analysis shows that while most trajectories correctly identify problematic files (72-81\% even in failures), success depends more on achieving approximate rather than exact code modifications. These and other findings unveiled by our study, provide a foundation for understanding agent behaviour through trajectory analysis, contributing to the development of more robust and interpretable autonomous software engineering systems.

代码生成智能体轨迹分析LLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。