提出可定位、归因并修复长时搜索代理失败的审计框架
SearchAuditor: Auditing and Attributing Failures in Long-Horizon Search Agents

- 通过多视角证据仲裁实现对搜索错误的精准定位
- 在8个模型上平均73条消息的失败轨迹中实现32.3%的修复成功率
- 适合研究大模型搜索系统鲁棒性与自动化调试的研究者
深度搜索代理通过长时间的网络交互解决复杂问题,这一过程既复杂又脆弱:微小推理错误可能在长而嘈杂的轨迹中传播,导致看似流畅但错误的答案。诊断此类失败需手动检查极长的执行轨迹,超出人类处理能力。为此,我们提出SearchAuditBench基准,评估大模型审计器是否能定位、归因并修复这些失败,从而减轻人工负担。该基准包含1,243条失败轨迹,平均73.1条消息和65.1K tokens,来自八个开源模型在五个深度搜索基准上的表现,每条均经专家标注关键错误步骤、搜索特异性根因及参考修复方案,并附评分标准。我们进一步提出SearchAuditor,一种基于证据的多视角审计框架,有效实现失败的定位、归因与修复。实验表明,即使最强基线在使用GPT-5.5等前沿模型时,端到端通过率也仅达26.6%。相比之下,SearchAuditor在不同前沿模型下均优于所有基线,端到端通过率达32.3%,且其修复方案可帮助代理更好恢复错误状态。
原文摘要 · Abstract (English)
Deep search agents tackle challenging questions through long-horizon web interactions, a process that is both complex and fragile: small reasoning errors may propagate through long, noisy trajectories into fluent but incorrect answers. Diagnosing such failures is difficult, requiring the manual inspection of extremely long execution traces, which could be beyond human capacity. We therefore introduce SearchAuditBench, a benchmark that evaluates whether LLM auditors can localize, attribute, and repair these failures, thereby reducing the human burden. SearchAuditBench comprises 1,243 failed trajectories, averaging 73.1 messages and 65.1K tokens, collected from eight open-weight models on five deep-search benchmarks, each expert-annotated with the critical error step, a search-specific root cause, and a reference repair with grading rubrics. We further propose SearchAuditor, a multi-perspective auditing framework that effectively localizes, attributes, and repairs search-agent failures through evidence-grounded adjudication. Experimental results show that even the strongest baseline, when powered by a frontier model like GPT-5.5, attains only a 26.6% end-to-end pass rate. In contrast, our SearchAuditor consistently outperforms all baselines across different frontier models, achieving an end-to-end pass rate of 32.3%, and resuming failed runs with its repairs enables agents to better recover from errors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。