arXiv:2506.18824cs.SEcs.AI2025-06中稿 · ASE 2025被引 81

分析代码修复代理的思考-动作-结果轨迹,揭示成功与失败模式

Understanding Software Engineering Agents: A Study of Thought-Action-Result Trajectories

  • 统一三种顶尖代码代理的交互日志,构建标准化轨迹数据集
  • 发现平均迭代次数达12次,高成功率执行更注重反馈整合
  • 提供可复用的标注框架,助力透明化智能开发工具设计

基于大语言模型(LLM)的智能体正被广泛用于程序修复和问题解决等复杂软件工程任务。这些智能体通过自主生成自然语言思考、调用外部工具并迭代优化解决方案来工作。尽管应用广泛,其内部决策过程仍不清晰,限制了对其运行机制和失效模式的理解。本文对三种先进代理——RepairAgent、AutoCodeRover 和 OpenHands——的思考-动作-结果轨迹进行大规模实证研究。我们统一其交互日志,收集了120条完整轨迹和2,822次LLM交互,聚焦于程序修复与问题解决任务。研究结合定量分析(结构特征、动作模式、令牌消耗)与定性评估(推理连贯性、反馈融合),识别出关键轨迹特征:平均迭代12次,常见动作序列,以及思想、行动与结果间的语义一致性。研究揭示了区分成功与失败执行的行为模式与反模式,为改进提示策略、故障诊断与反模式检测提供了实用洞见。我们公开发布数据集与标注框架,以支持透明且稳健的自主软件工程智能体研究。

原文摘要 · Abstract (English)

Large Language Model (LLM)-based agents are increasingly employed to automate complex software engineering tasks, such as program repair and issue resolution. These agents operate by autonomously generating natural language thoughts, invoking external tools, and iteratively refining their solutions. Despite their widespread adoption, the internal decision-making processes of these agents remain largely unexplored, limiting our understanding of their operational dynamics and failure modes. In this paper, we present a large-scale empirical study of the thought-action-result trajectories of three state-of-the-art LLM-based agents: RepairAgent, AutoCodeRover, and OpenHands. We unify their interaction logs into a common format, capturing 120 trajectories and 2,822 LLM interactions focused on program repair and issue resolution. Our study combines quantitative analyses of structural properties, action patterns, and token usage with qualitative assessments of reasoning coherence and feedback integration. We identify key trajectory characteristics, such as iteration counts and token consumption, recurring action sequences, and the semantic coherence of thoughts, actions, and their results. Our findings reveal behavioral motifs and anti-patterns that distinguish successful from failed executions, providing actionable insights for improving agent design, including prompting strategies, failure diagnosis, and anti-pattern detection. We release our dataset and annotation framework to support further research on transparent and robust autonomous software engineering agents.

智能代理代码修复行为分析轨迹研究

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。