arXiv:2608.22856cs.IRcs.CL2026-08

检索增强问答系统更新索引后答案会无故改变,需审计兼容性而非仅看准确率。

Same Agent, Different Answers: A Repeat-Aware Audit of Corpus-Induced Answer Churn in Retrieval-Augmented QA

  • 提出快照兼容性审计,分离出索引更新引发的异常答案漂移
  • 在Natural Questions上发现语义漂移高达10.25个百分点,准确率却仅降1.5点
  • 适合关注模型稳定性与部署可靠性的研究者和工程师

检索增强型问答系统在索引扩展后,即使模型标识、提示、检索策略等保持不变,仍可能返回不同答案。整体准确率可能掩盖这种变化,而普通生成波动又会夸大更新影响。我们称此为‘隐蔽的答案漂移’,并提出‘快照兼容性审计’方法,通过减去同快照重复测试的差异来估算超额答案漂移。以冻结的FineWeb前缀从1个分片扩展到7个分片为例,在预注册的400题Natural Questions实验中,标准化精确匹配与盲化语义的超额漂移分别为6.44和10.25个百分点,而精确匹配准确率仅下降1.50点。后验分析发现40/400题出现重复稳定但语义翻转的情况。另一次预注册的200题TriviaQA实验显示较小但方向一致的超额漂移,准确率变动方向相反。在另一组不依赖结果的100题复现实验中,使用第二个DeepSeek生成器与配置,仍观测到8.75个百分点的语义漂移,而精确匹配上升3.00点。因此,答案兼容性可能失效而不伴随明显或一致的性能变化。检索增强发布应同时审计兼容性与效用。

原文摘要 · Abstract (English)

A retrieval-augmented QA system can return different answers after an index expansion even when its requested model identifier, prompt, retrieval policy, evidence depth, rendering, and exposed generation controls are held fixed. Aggregate accuracy may hide these changes when gains and losses cancel, while ordinary generation variability makes one-shot comparisons overstate update effects. We call the hidden phenomenon accuracy-blind answer churn and introduce the \emph{Snapshot Compatibility Audit}, which estimates excess answer churn by subtracting same-snapshot repeat disagreement from cross-snapshot disagreement. We instantiate it by expanding one frozen FineWeb prefix from one to seven shards. In a preregistered 400-question Natural Questions study, normalized-exact and blinded-semantic excess churn are 6.44 and 10.25 percentage points while exact-match accuracy changes by only $-1.50$ points. A post-hoc analysis finds repeat-stable semantic flips on 40/400 questions. A separately preregistered 200-question TriviaQA study yields smaller, directionally consistent excess churn while exact-match accuracy moves in the opposite direction. An outcome-blind post-hoc 100-question subset replication with a second DeepSeek generator and serving configuration finds 8.75 pp of semantic excess churn even as exact match rises by 3.00 percentage points. Answer-level compatibility can therefore fail without a conspicuous or consistently directed utility shift. Retrieval-augmented releases should audit compatibility alongside utility.

问答系统索引更新答案漂移模型审计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。