单纯追求相关性会限制大模型回答质量，新方法通过多标准重排序提升性能。
Relevance Isn't All You Need: Scaling RAG Systems With Inference-Time Compute Via Multi-Criteria Reranking
- 引入多标准重排序机制，结合相关性和答案质量优化检索结果
- 在增加推理时间计算量时，传统方法表现不佳，新方法显著提升效果
- 适用于需要高精度回答的场景，尤其适合长周期推理任务
现代大语言模型系统普遍依赖检索增强生成（RAG），其目标是获取对回答生成有帮助的上下文。现有RAG系统通常仅以最大化上下文与查询的相关性为优化目标。然而，传统理论表明，仅追求相关性而忽略其他显式标准可能导致信息瓶颈。我们在当代LLM环境下重新验证了这一结论：在标准RAG流程中，仅最大化相关性反而会降低下游回答质量。为此，我们评估了同时考虑上下文相关性和答案质量的现有RAG方法，发现这些方法在增加推理时间计算资源时，表现无法有效提升。为此，我们提出「RErank BEyond reLevance (REBEL)」，通过链式思维提示（Chain-of-Thought prompting）及可选多轮对话注入多标准优化，使RAG系统能随推理时间计算量增长而持续提升性能。该方法实现了新的性能/速度权衡曲线，使检索相关性与回答质量同步提高。代码可在llama-index PR #17590 及 Microsoft REBEL 仓库中获取。
原文摘要 · Abstract (English)
Modern Large Language Model (LLM) systems typically rely on Retrieval Augmented Generation (RAG) which aims to gather context that is useful for response generation. These RAG systems typically optimize strictly towards retrieving context that is maximally relevant to the query. However, conventional theory suggests that retrieval systems which seek to maximize context relevance without any additional explicit criteria can create information bottlenecks. We reaffirm this finding in the modern age of LLM's by showing that in standard RAG pipelines, maximizing for context relevance alone can degrade downstream response quality. In response, we show evaluations of existing RAG methods which account for both context relevance and answer quality. These evaluations introduce a novel finding that existing RAG systems scale poorly with inference time compute usage when considering our combined metric. We introduce "RErank BEyond reLevance (REBEL)", which enables RAG systems to scale with inference-time compute via injection of multi-criteria optimization using Chain-of-Thought prompting (and optionally Multi-Turn dialogue). Ultimately, this enables a new performance/speed tradeoff curve, where RAG systems are able to achieve both higher relevance of retrieved contexts and superior answer quality as inference time increases. Code for the implementation of our method in llama-index can be found at the following PR: https://github.com/run-llama/llama_index/pull/17590. Code for running experiments using this llama-index implementation can be found at https://github.com/microsoft/REBEL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。