重新验证RAG中文档位置与上下文长度的影响,发现结果受主题数量和检索质量显著干扰。
Lost in the Evidence? Reproducing Document Position and Context Size Effects in RAG
- 通过控制实验框架,发现小主题集会掩盖或夸大位置效应
- 在稳定主题集上复现了现代大模型的'中间丢失'现象及位置偏差
- 强调真实检索场景下评估方法对结论影响巨大,适合做RAG研究者参考
检索增强生成(RAG)系统依赖将检索到的文档拼接进模型输入上下文,因此文档排序和上下文大小是关键但有争议的设计选择。先前研究报道了基于位置的效应,如“中间丢失”等长上下文现象,但实证结果在不同模型、数据集和评估协议间不一致且难以复现。本文开展系统性可复现性研究,重新审视这些结论,并在受控评估框架下考察其在当代大语言模型中的演变。我们首先指出主题采样是主要变异性来源:小主题集可能掩盖或夸大排序效应。基于多轮子集采样,提出一种实用校准流程,识别出在合理成本下产生稳定趋势的主题数量。使用这些固定主题集,我们复现并扩展了位置敏感性的结果,重新评估现代大模型中的‘中间丢失’和位置偏差。此外,我们研究更真实的RAG场景——相关性由检索器决定而非理想化的真值文档。在此设置下,我们重审一项近期产业研究,发现评估选择如主题覆盖有限和依赖大模型裁判导致结果偏差。最后,我们分析了在不完美检索条件下,检索顺序与上下文大小如何影响下游大模型性能。结果表明,两者与检索质量及模型选择强烈交互,理想化设定下的结论未必适用于真实RAG流水线。我们发布全部代码与配置,以支持可复现性和未来鲁棒性RAG评估研究。
原文摘要 · Abstract (English)
Retrieval-Augmented Generation (RAG) systems rely on retrieved documents being concatenated into a model's input context, making both document ordering and context size critical yet controversial design choices. Prior work reports position-based effects such as lost in the middle and related long-context phenomena. However, empirical findings remain inconsistent and hard to reproduce across models, datasets, and evaluation protocols. In this paper, we present a systematic reproducibility study that revisits these claims and examines how they evolve with contemporary LLMs under a controlled evaluation framework. We first show that topic sampling is a major source of variance: small topic sets can mask or exaggerate ordering effects. Based on repeated subset sampling across multiple topic budgets, we provide a practical calibration procedure that identifies topic counts yielding stable trends at feasible cost. Using these fixed topic sets, we then reproduce and extend results on position sensitivity, re-evaluating lost in the middle and positional biases in modern LLMs. Then, we also study a more realistic RAG scenario in which relevance is mediated by a retriever rather than oracle access to ground-truth documents. In this setting, we re-examine a recent industry study and identify discrepancies to evaluation choices such as limited topic coverage and reliance on LLM-based judges. Finally, we conduct an analysis of how retrieval order and context size affect downstream LLM performance under imperfect retrieval. Our results demonstrate that both factors interact strongly with retrieval quality and model choice, and that conclusions drawn from idealised setups do not always transfer to real-world RAG pipelines. We release all code and configurations to support reproducibility and future work on robust RAG evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。