噪声能提升问答性能?研究发现这效果极依赖实验设置。
The Powerless Noise: How Experimental Settings Shape the Reported Power of Noise

- 通过复现与扩展实验,检验噪声提升效果的稳定性。
- 小改动提示词或解码限制,噪声效应就消失或减弱。
- 提醒研究者警惕推理设计对结果的影响,避免误判。
近期研究提出,在检索增强生成(RAG)系统中加入无关文档可提升问答表现,称为'噪声之力'。本文复现了Cuconasu等人的主要结论,并在扩展实验设置下评估该现象的鲁棒性。首先确认在原始设置(早期LLM、受限提示和解码策略)下该现象成立。随后通过一系列消融实验,考察模型选择、指令提示和输出长度约束等设计因素的影响。结果显示,'噪声之力'对推理配置极为敏感:微调提示方式或解码限制,该效应可能显现、减弱甚至消失。结合错误分析发现,截断和生成格式错误贡献显著。因此,在当前实验条件下,无法稳健证实噪声检索具有普遍益处。本工作强调了在RAG系统中仔细审查推理设计的重要性。代码已开源。
原文摘要 · Abstract (English)
Recent work has suggested that adding irrelevant documents to the input of retrieval-augmented generation (RAG) systems can improve question-answering performance, a phenomenon referred to as the Power of Noise. This motivated investigations into the role of noise in information retrieval. In this paper, we reproduce the main findings of Cuconasu et al. and evaluate the robustness of the effect under extended experimental settings. We first confirm that the phenomenon holds under the original setup, which uses earlier-generation LLMs, restrictive prompting and constrained decoding settings. We subsequently introduce a series of extensions to investigate the underlying causes of the noise effect, examining the authors' original design choices including the use of different models, instruction prompting, and relaxed output length constraints. Across these ablations, the Power-of-Noise pattern proves highly sensitive to inference configuration: it can appear, weaken, or disappear under small changes to prompt formulation and decoding limits. Combined with our error analysis, which shows substantial contributions from truncation and malformed generations, this variance indicates that the original effect cannot be robustly confirmed as a general benefit of noisy retrieval under these experimental conditions. More broadly, our work highlights the importance of carefully scrutinizing inference design in retrieval-augmented generation systems. Our code is available at https://github.com/ina0105/The-Power-of-Noise-Reproduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。