通过排序组合检索结果提升生成质量，缓解模型偏见问题。
Inference Scaling for Bridging Retrieval and Augmented Generation
- 用多种检索片段顺序的多次推理聚合，减少生成偏差。
- 在MS MARCO和HotpotQA上分别提升约7点ROUGE-L和EM指标。
- 能利用检索器先验知识降低计算开销，适合高效RAG场景。
检索增强生成（RAG）通过将检索到的上下文作为输入来引导大语言模型（LLM）输出。然而，现有研究发现生成模型存在偏见：提升检索效果反而可能损害最终生成结果。本文提出一种基于推理放大的方法——干预混合（Mixture-of-Intervention, MOI），通过对检索片段的不同排列进行多次前向推理，显式建模每段内容的去偏效用，并构建新排名。MOI还能利用检索器的先验知识，减少需考虑的排列数量，降低每次LLM调用的开销。在多个RAG任务中验证有效，使MS MARCO上的ROUGE-L和HotpotQA上的EM分别提升约7个百分点。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) has emerged as a popular approach to steering the output of a large language model (LLM) by incorporating retrieved contexts as inputs. However, existing work observed the generator bias, such that improving the retrieval results may negatively affect the outcome. In this work, we show such bias can be mitigated, from inference scaling, aggregating inference calls from the permuted order of retrieved contexts. The proposed Mixture-of-Intervention (MOI) explicitly models the debiased utility of each passage with multiple forward passes to construct a new ranking. We also show that MOI can leverage the retriever's prior knowledge to reduce the computational cost by minimizing the number of permutations considered and lowering the cost per LLM call. We showcase the effectiveness of MOI on diverse RAG tasks, improving ROUGE-L on MS MARCO and EM on HotpotQA benchmarks by ~7 points.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。