让复杂智能体执行日志可读可诊断,自动生成精准分析报告
TraceSIR: A Multi-Agent Framework for Structured Analysis and Reporting of Agentic Execution Traces
- 三代理协同:压缩日志、精确诊断、生成报告
- 在三个真实场景中报告质量显著优于现有方法
- 适合调试智能体系统或开发自动化运维工具的工程师
智能体系统通过调用外部工具和迭代决策完成深度研究、函数调用和编程等复杂任务,但其长而复杂的执行日志使得故障诊断和根因分析极为困难。手动检查难以扩展,直接用大模型分析原始日志又受限于输入长度且推理不可靠。仅关注最终任务结果还会丢弃关键行为信息。为此,我们提出 TraceSIR,一种用于结构化分析与报告智能体执行日志的多代理框架。TraceSIR 协调三个专用代理:(1) StructureAgent,引入新型抽象格式 TraceFormat,压缩执行日志同时保留核心行为信息;(2) InsightAgent,实现细粒度诊断,包括问题定位、根因分析及优化建议;(3) ReportAgent,聚合多个任务实例的洞察并生成综合分析报告。为评估 TraceSIR,我们构建了 TraceBench,涵盖三个真实世界智能体场景,并提出 ReportEval 评估协议,用于衡量报告质量与工业需求的一致性。实验表明,TraceSIR 持续生成连贯、信息丰富且可操作的报告,在所有评估维度上均显著优于现有方法。项目与视频已公开于 https://github.com/SHU-XUN/TraceSIR。
原文摘要 · Abstract (English)
Agentic systems augment large language models with external tools and iterative decision making, enabling complex tasks such as deep research, function calling, and coding. However, their long and intricate execution traces make failure diagnosis and root cause analysis extremely challenging. Manual inspection does not scale, while directly applying LLMs to raw traces is hindered by input length limits and unreliable reasoning. Focusing solely on final task outcomes further discards critical behavioral information required for accurate issue localization. To address these issues, we propose TraceSIR, a multi-agent framework for structured analysis and reporting of agentic execution traces. TraceSIR coordinates three specialized agents: (1) StructureAgent, which introduces a novel abstraction format, TraceFormat, to compress execution traces while preserving essential behavioral information; (2) InsightAgent, which performs fine-grained diagnosis including issue localization, root cause analysis, and optimization suggestions; (3) ReportAgent, which aggregates insights across task instances and generates comprehensive analysis reports. To evaluate TraceSIR, we construct TraceBench, covering three real-world agentic scenarios, and introduce ReportEval, an evaluation protocol for assessing the quality and usability of analysis reports aligned with industry needs. Experiments show that TraceSIR consistently produces coherent, informative, and actionable reports, significantly outperforming existing approaches across all evaluation dimensions. Our project and video are publicly available at https://github.com/SHU-XUN/TraceSIR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。