自动分析大模型代理执行日志，发现系统性错误模式。
Insights Generator: Systematic Corpus-Level Trace Diagnostics for LLM Agents
- 构建多智能体系统，自动提出并验证诊断假设。
- 专家使用报告后，基础框架性能提升30.4个百分点。
- 适合需要可靠调试的LLM应用开发与运维人员。
LLM代理的故障诊断仍主要依赖人工。从业者仅检查少量执行轨迹，形成临时假设并迭代，难以发现跨轨迹群体的规律，也无法扩展到包含数万标记符的生产级数据。本文形式化了轨迹语料库层面的诊断问题：给定一组执行轨迹，目标是生成基于自然语言、可支撑证据的洞察，描述不同轨迹组中的系统性行为模式。我们提出Insights Generator（IG），一种多智能体系统，通过在轨迹语料库中提出和测试假设，回答诊断问题，并生成有证据支持的洞察报告。在定性和定量维度评估显示，使用IG报告的人类专家使基础框架性能提升30.4个百分点，优于未修改基线；利用IG洞察的编码代理也表现出一致且稳定的性能增益。在多个基准上，IG的侦察-调查者架构在检测覆盖率上与现有方法相当，领域专家评价其报告在深度和证据质量上表现领先。
原文摘要 · Abstract (English)
Diagnosing failures in LLM agents remains largely manual. Practitioners inspect a small subset of execution traces, form ad-hoc hypotheses, and iterate. This process misses patterns that only emerge across trace populations and does not scale to production corpora where individual traces span tens of thousands of tokens. We formalize the problem of corpus-level trace diagnostics. Given a corpus of execution traces, the goal is to produce grounded natural-language insights that characterize systematic behavioral patterns across trace groups, each linked to supporting evidence. We present the Insights Generator (IG), a multi-agent system that answers diagnostic questions by proposing and testing hypotheses across the trace corpus to produce an evidence-backed insights report. We evaluate IG across qualitative and objective dimensions, spanning rubric-based report assessment and downstream performance improvements achieved by implementing IG insights. Human experts using IG reports improve scaffold performance by 30.4pp over the unmodified baseline scaffold, and coding agents leveraging IG-derived insights show consistent and stable gains. Across benchmarks, IG's scout-investigator architecture produces findings comparable in detection coverage to competing approaches, while domain experts rated IG reports as leading depth and evidence quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。