arXiv:2609.06147cs.CLcs.AI2026-09

发现文档问答中因信息缺失导致模型答案不稳定,提出可复现的审计方法。

What the Window Does Not Contain: Auditing Provenance in a Document-Grounded Instability Benchmark

  • 构建60任务470项的真实融资文件基准,检测模型回答波动
  • 36项因证据缺失导致答案波动率达0.255,远高于干净数据的0.087
  • 提供可运行的审计工具,助开发者排查文档基准缺陷

让语言模型对同一文档重复提问20次,其答案常不一致。我们构建了包含60个任务和470个样本的真实风险投资文件基准Probity,用于衡量此类不稳定性。审计发现,任何基于文本片段的基准都可能携带缺陷:部分问题的证据未包含在模型看到的文本窗口中。该审计识别出36个问题,将两类失败区分开:证据确实缺失,以及需从窗口内数值计算得出的答案。被标记项的答案波动率高达0.255,显著高于其余427个无缺陷样本的0.087;剔除这些项后,跨模型一致性下降约五分之一。我们预判:若重新裁剪窗口以包含证据,不稳定性应低于阈值——但修复仅使波动降低0.058,置信区间包含零,故结论为相关性而非因果。几乎所有测量结果均无法体现不稳定性,这限制了以准确率为目标的语料库对稳定性的评估能力。我们公开语料、全部112,800条原始响应及审计脚本,供任意文档基准使用。

原文摘要 · Abstract (English)

Ask a language model the same question about the same document twenty times, and it sometimes returns two different answers. We built Probity, a benchmark of 60 tasks and 470 items from real venture-financing filings, to measure how often this happens. Then we audited our own corpus and found a defect any excerpt-built benchmark can carry: items whose evidence is missing from the window of text the model is shown. The audit flags 36 items and separates two failures a single flag would conflate: evidence genuinely absent from the window and answers that must be computed from numbers the window does supply. Flagged items change their answers far more often, wobbling at 0.255 against 0.087 on the 427 clean items, and excluding them cuts apparent cross-model agreement by about a fifth. Before testing whether the missing evidence explains the instability, we registered a prediction: re-cut each window to hold its evidence, and instability should fall below a set threshold. It failed: the repair moved wobble by 0.058, with an interval containing zero. We report the association as correlational. Almost all measurements sit where instability cannot show, which bounds what a corpus built for accuracy can say about stability. We release the corpus, all 112,800 raw responses, and the audit as a runnable check for any document benchmark.

模型稳定性文档问答基准测试审计工具

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。