提出新基准,追踪智能体检索生成中错误传播路径
When Failures Propagate: Causal Failure Attribution in Agentic Retrieval-Augmented Generation
- 通过注入故障并重执行,定位错误源头的传播路径
- 第一跳错误诊断准确率91%,后续跳骤降至0%
- 适合研究大模型推理可靠性与故障溯源的学者
智能体检索增强生成(Agentic RAG)在多跳推理中交替进行检索、推理和生成。第一跳的检索错误可能仅在第三跳表现为错误答案,而后续检索也可能修复该路径。本文提出AgenticRAG-FP,一个干预式基准,用于因果故障归因。该基准在指定跳跃处注入已验证故障,重新执行下游轨迹,并评估诊断器对已知干预的识别能力。核心问题是:在后缀变化后,事后追踪是否仍能识别注入故障的跳跃。在80个三跳MuSiQue问题上完成的严格密集Claude Haiku 4.5测试中,基于覆盖度的诊断在第一跳准确率为0.91,在第二跳和第三跳分别为0.00(失败轨迹数分别为43、36、21)。在较小规模的内容破坏研究中,对主题一致证据中的答案相关或桥接事实进行修改。在深度2下,过滤后剩余18个失败案例,覆盖度诊断为0.00,固定跳跃反事实探针在探索性合并比较中为0.67。深度3内容估计仅为描述性,因仅有三个失败案例。结果表明,传播深度应成为诊断智能体RAG失败的显式评估轴,同时区分事后信号丢失与小样本方法比较。
原文摘要 · Abstract (English)
Agentic retrieval-augmented generation (RAG) interleaves retrieval, reasoning, and answer generation across multiple hops. A retrieval error at hop 1 can surface only as a wrong answer at hop 3, while later retrieval can also repair the trajectory. This paper introduces AgenticRAG-FP, an interventional benchmark for causal failure attribution in agentic RAG. The benchmark injects a certified fault at a specified hop, re-executes the downstream trajectory, and evaluates diagnosers against the known intervention. Its central question is whether a post-hoc trace still identifies the injected hop after the suffix changes. In the completed strict dense Claude Haiku 4.5 sweep on 80 three-hop MuSiQue questions, coverage-based diagnosis is 0.91 at hop 1 and 0.00 at hops 2 and 3 (n=43,36,21 failed trajectories). A smaller content-corruption study changes an answer-bearing or bridge fact in topically intact evidence. At depth 2, where 18 failed cases remain after filtering, coverage-based diagnosis is 0.00 and a frozen-hop counterfactual probe is 0.67 in an exploratory pooled comparison. Depth-3 content estimates are descriptive only because they contain three failed cases. These results make propagation depth an explicit evaluation axis for diagnosing agentic RAG failures while distinguishing broad evidence of post-hoc signal loss from small-sample method comparisons.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。