用图结构分析智能软件系统的执行过程,揭示其决策逻辑与效率问题。
Process-Centric Analysis of Agentic Software Systems
- 构建Graphectory图模型,捕捉智能系统执行中的时序与语义关系
- 4000条轨迹分析显示:强模型和好提示带来更复杂、深入的探索行为
- 实时监测并干预异常轨迹,提升问题解决率6.9%-23.5%,几乎无额外开销
智能软件系统由协同模块组成,其执行过程具有随机性和自适应性,传统评估只关注结果而忽略过程细节。我们提出Graphectory,将这类系统的动态行为建模为带时序和语义关系的图结构。基于此,我们自动分析了两个主流智能编程工作流(SWE-agent与OpenHands)在四种大型语言模型下的4000条执行轨迹,以解决SWE-bench问题。结果显示:使用更丰富提示或更强模型的代理展现出更复杂的Graphectory,体现更深入的探索与验证;问题难度与底层模型影响策略——成功案例多呈定位-修复-验证的连贯流程,失败案例则呈现混乱或回溯行为;即使成功系统也常存在效率低下问题。我们进一步实现一种实时构建与分析Graphectory及Langutory的技术,在执行中检测异常轨迹并触发诊断消息或回滚操作。实验表明,该在线监控与干预机制使问题实例的解决率提升6.9%-23.5%,显著缩短轨迹长度且开销近乎为零。
原文摘要 · Abstract (English)
Agentic systems are modern software systems: they consist of orchestrated modules, expose interfaces, and are deployed in software pipelines. Unlike conventional programs, their execution, i.e., trajectories, is inherently stochastic and adaptive to the problems they solve. Evaluation of such systems is often outcome-centric. This narrow focus overlooks detailed insights, failing to explain how agents reason, plan, act, or change their strategies. Inspired by the structured representation of conventional software systems as graphs, we introduce Graphectory to systematically encode the temporal and semantic relations in such systems. Using Graphectory, we automatically analyze 4000 trajectories of two dominant agentic programming workflows, SWE-agent and OpenHands, with four backbone Large Language Models (LLMs), attempting to resolve SWE-bench issues. Our automated analyses (completed within four minutes) reveal that: (1) agents using richer prompts or stronger LLMs exhibit more complex Graphectory, reflecting deeper exploration, broader context gathering, and more thorough validation; (2) agents' strategies vary with problem difficulty and the underlying LLM - for resolved issues, strategies often follow coherent localization-patching-validation steps, while unresolved ones exhibit chaotic or backtracking behaviors; and (3) even successful agentic systems often display inefficient processes. We also implement a novel technique for real-time construction and analysis of Graphectory and Langutory during agent execution to flag trajectory issues. Upon detecting such issues, the technique notifies the agent with a diagnostic message and, when applicable, rolls back the trajectory. Experiments show that online monitoring and interventions improve resolution rates by 6.9%-23.5% across models for problematic instances, while significantly shortening trajectories with near-zero overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。