arXiv:2608.29477cs.CLcs.AI2026-08中稿 · the Context Beyond…

区分长文本与干扰项对文档问答的影响,发现真实干扰项更难处理。

MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects

论文配图:MUDDLE: Measuring Understanding of Documents under Distractor and Length Effects
图 1 · 摘自论文原文
  • 设计可控实验分离长度与主题干扰影响,用真实问题验证模型鲁棒性。
  • gpt-5-mini在含主题相似干扰项时准确率显著下降,远低于等长随机干扰。
  • 结果表明模型对语义相关干扰更敏感,适合评估真实场景下的问答系统。

文档问答系统越来越多地在检索到的文档集合上回答问题,而非单一清晰来源,因此对干扰内容的鲁棒性与阅读能力同样重要。以往研究常混淆长文本与主题相近干扰项的影响。本文提出MUDDLE,一个受控基准,可分离这两种效应。使用270个人工标注的问题,每个问题对应单一源文档,并在五种条件下实例化:仅源文档、源文档加两个或四个主题相似的难负样本、源文档加两个或四个随机干扰项。随机干扰项在长度和来源上与难负样本匹配,因此准确率差异反映主题相似性而非长度。所有条件以markdown、页面图像和原始PDF形式呈现,但本次报告使用markdown版本,因包含干扰项的输入超出当前图像和PDF处理限制。答案由LLM评判器在三个模型家族中评分。在完整的markdown测试中,对于gpt-5-mini,主题相似干扰项在两种上下文规模下均比等长随机文档导致更低准确率,而随机文档接近无干扰基线。该效应虽小但方向一致,且当跨上下文规模合并时,主题相似干扰项显著劣于等长随机干扰项。数据与评估代码已发布,支持可复现的上下文退化研究。

原文摘要 · Abstract (English)

Document question-answering systems increasingly answer questions over collections of retrieved documents rather than one clean source, so robustness to distracting context matters as much as reading ability. When such systems fail, it is often unclear whether the context was too long or the distractors were too close to the topic, because prior work tends to conflate these two effects. We present MUDDLE, a controlled benchmark that separates them. MUDDLE uses 270 human-annotated questions, each tied to a single source document, and instantiates every question in five conditions: the source alone, the source with two or four topically similar hard negatives, and the source with two or four random distractors. The random distractors are matched to the hard negatives in length and provenance, so an accuracy gap between the two arms reflects topical similarity rather than length. All five conditions are rendered in markdown, page images, and raw PDF, but the distractor sweep reported here is run in markdown, since a source plus its distractors exceeds current image and PDF input limits. We score answers with an LLM judge across three model families. In the complete markdown sweep, hard negatives lower accuracy more than length-matched random documents at both context sizes for gpt-5-mini, while random documents stay near the no-distractor baseline. The effect is small but directionally consistent, and for gpt-5-mini hard negatives significantly underperform length-matched random distractors when pooled across context sizes. We release the data and evaluation code for a reproducible study of context degradation.

文档问答干扰项模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。