通过对照实验诊断RAG失败原因，发现修复证据可挽救部分错误回答。
What Would Fix This RAG Failure? Auditing Counterfactual Response with Paired Evidence Interventions
- 设计配对干预实验，分别添加缺失证据和删除错误证据，测试响应变化。
- 添加支持证据修复600个联合案例中的197例（32.8%），删除非支持证据修复162例（13.6%）。
- 结果表明模型对证据敏感但不可完全预测，适合研究RAG鲁棒性的学者参考。
一次RAG生成失败的回答可能与多个未见的证据修复结果一致。本文提出Pair-ID，一种离线审计方法：固定查询、检索状态和阅读器，交叉执行两项操作——添加缺失支持与删除已验证的非支持内容，以测量同一失败情况下的反事实响应向量。在19,981个基准查询上进行完整漏斗分析，识别出11,105个可修复的Qwen失败案例，再通过预设的SHA-256排序筛选出1,200个，生成任何采样响应前完成筛选。在1,190个再生有效的失败中，支持添加修复了600个联合案例中的197例（0.328，95% CI [0.292, 0.367]），删除操作修复1,190例中的162例（0.136，95% CI [0.117, 0.155]）；长度与位置匹配的伪样本保留语义差异分别为0.223和0.101。原始观察具备一定预测能力（宏观AUROC 0.678；Brier得分0.152对比基线0.160），但向量精确率0.637低于多数向量基线0.646，宏F1仅为0.170。跨四种阅读器，边际敏感性重复出现，整体向量一致性为0.675–0.765，仅联合案例下降至0.538–0.691。结果表明，证据敏感性在哈希筛选的可修复失败样本中显著存在，且对原失败响应的预测有限，依赖具体阅读器。研究支持一种基于框架的离线响应审计，而非信息论不可能结论、读者无关分类或运行时修复策略。
原文摘要 · Abstract (English)
A failed retrieval-augmented generation (RAG) answer can be consistent with several unseen responses to evidence repair. We introduce Pair-ID, an offline audit that holds one query, retrieval state, and reader constant, then crosses two operations, adding missing support and deleting verified nonsupport, to measure a same-failure counterfactual response vector. A complete funnel over 19,981 benchmark queries identifies 11,105 eligible Qwen failures, from which a prospectively fixed SHA-256 ordering selects 1,200 before generating any sampled response. Among 1,190 regenerated-valid failures, support addition repairs 197/600 JOINT cases (0.328, 95% CI [0.292, 0.367]), and deletion repairs 162/1,190 cases (0.136, 95% CI [0.117, 0.155]); length- and position-matched shams retain semantic contrasts of 0.223 and 0.101. The original view carries partial predictive signal for individual response cells (macro AUROC 0.678; Brier 0.152 versus 0.160 for a marginal baseline), but exact-vector accuracy, 0.637, does not exceed the 0.646 majority-vector baseline, and vector macro-F1 is 0.170. Across four readers, both marginal sensitivities recur, while pooled exact-vector agreement is 0.675-0.765 and JOINT-only agreement falls to 0.538-0.691. These results show that evidence sensitivity occurs at meaningful rates in the hash-selected eligible-failure sample, is only partially predictable from the observed failure, and is conditional on the reader. The evidence supports a frame-scoped offline response audit, not an information-theoretic impossibility result, reader-independent taxonomy, or runtime repair policy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。