通过异常感知可观测性，提前发现多智能体大模型中的无效计算
Early Diagnosis of Wasted Computation in Multi-Agent LLM Systems via Failure-Aware Observability
- 基于追踪构建在线信号，识别循环、预算压力等异常模式
- 警告后仍消耗58.1%的计算资源，说明干预窗口期长
- 用预警信息引导搜索多样性，可将后续耗能降至30.4%
失败感知可观测性可在最终答案评估前诊断多智能体大模型系统中的无效计算。我们提出一种基于追踪的三智能体框架——协调器、搜索代理和执行代理，将结构化事件转化为在线信号，用于检测循环、预算压力、信息增益低和工具不稳等问题，并引入离线语义根基度量与选择性大模型作为裁判的评估。在165个GAIA验证轨迹上，相同约束下98次产生有效答案，67次失败或未生成答案。在被警告的失败运行中，平均58.1%的令牌在首次警告后消耗，表明干预有巨大空间。一个包含10个任务的二级试点项目利用警告信息实现搜索多样化或要求证据，使基线中警告后的令牌占比从0.638降至0.304。结果支持分层设计：低成本在线信号帮助协调器重定向或终止冗余行为，而更深层的语义检查则判断已完成答案是否具备足够的可信根基。
原文摘要 · Abstract (English)
Failure-aware observability diagnoses wasted computation in multi-agent LLM systems before final-answer evaluation can explain what went wrong. We propose a trace-based framework for a three-agent architecture -- orchestrator, search agent, and execution agent -- that converts structured events into online signals for loops, budget pressure, low information gain, and tool instability, then adds offline semantic grounding metrics and selective LLM-as-judge evaluation. On 165 GAIA validation traces under identical caps, 98 runs produce usable final answers and 67 fail or stop without one. Among warned failed runs, 58.1% of tokens are spent after the first warning on average, indicating substantial opportunity for intervention. A 10-task Level-2 pilot uses warnings to diversify search or require evidence, reducing post-warning token fraction from 0.638 in the baseline to 0.304. The results support a layered design: cheap online signals help the orchestrator redirect or halt redundant behavior, while deeper semantic checks identify whether completed answers are grounded enough to trust.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。