定位深度研究智能体错误源头，提升可信度分析能力
Where Do Deep-Research Agents Go Wrong? Span-Level Error Localization in Agent Trajectories
- 通过语义片段标注，构建可识别错误环节的评测基准
- 提出DRIFT框架，使错误定位准确率最高提升30个百分点
- 适合关注AI推理可靠性与可解释性的研究者使用
深度研究智能体通过搜索、工具调用、证据核查和答案合成等长轨迹完成任务。现有评估仅关注最终答案是否正确，无法判断哪些环节导致结果不可靠。本文收集来自两个智能体框架、三种主干模型和三个基准的数据集，共2,790条真实轨迹，将原始日志转化为语义片段，并通过大模型辅助专家评审标注有害错误片段。基于此构建了包含1,000个实例的TELBench基准，用于识别正常探索、失败搜索、暂定假设及无害噪声中的错误片段。进一步提出DRIFT框架——一种以主张为中心的审计机制，追踪智能体主张，检验其在轨迹中的证据支持，标记不支持或冲突主张影响答案路径的片段。跨模型家族与审计框架的实验表明，DRIFT可使片段级错误定位与首次错误识别准确率最高提升30个百分点。本工作为深度研究智能体的可靠性提供了过程层面的洞察。
原文摘要 · Abstract (English)
Deep-research agents solve tasks through long trajectories of search, tool use, evidence inspection, and answer synthesis. Evaluation based on final answers shows whether an agent succeeds, but not which parts of the trajectory make the answer unreliable. We study span-level error localization for deep-research agents. We collect 2,790 real trajectories from two agent frameworks, three backbone models, and three benchmarks, convert raw logs into semantic spans, and annotate harmful error spans through LLM-assisted expert review. From these annotations, we build TELBench, a 1,000-instance benchmark for identifying error spans among normal exploration, failed searches, tentative hypotheses, and harmless noise. We further propose DRIFT, a claim-centric auditing framework that tracks agent claims, checks their support in trajectory evidence, and marks spans where unsupported or conflicting claims affect the answer path. Experiments across model families and auditing frameworks show that DRIFT improves span-level error localization and first-error accuracy by up to 30 percentage points. Our work provides a process-level view of reliability in deep-research agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。