让大模型通过试错学习如何高效检索，减少幻觉。
Grounding by Trying: LLMs with Reinforcement Learning-Enhanced Retrieval
- 用强化学习让模型试不同搜索词，选出有效查询
- 检索准确率最高提升29%，生成效果提高17%
- 可适配任意现成检索器，适合通用大模型系统
大型语言模型（LLMs）的幻觉问题正通过允许其搜索真实信息源并基于这些来源生成回答来缓解。然而，面对复杂或间接话题时，LLMs常难以提出恰当的搜索查询。观察到模型可通过尝试不同查询并学习识别成功返回相关结果的查询来改进，我们提出LeReT（Learning to Retrieve by Trying），一种基于强化学习的检索优化框架，通过探索查询并使用偏好优化提升其质量。实验表明，LeReT可使检索准确率绝对提升高达29%，下游生成任务评估分数提升17%。该方法简单灵活，可适配任意现成检索器，是提升通用大模型流水线的有前景技术。项目网站：http://sherylhsu.com/LeReT/
原文摘要 · Abstract (English)
The hallucinations of large language models (LLMs) are increasingly mitigated by allowing LLMs to search for information and to ground their answers in real sources. Unfortunately, LLMs often struggle with posing the right search queries, especially when dealing with complex or otherwise indirect topics. Observing that LLMs can learn to search for relevant facts by $\textit{trying}$ different queries and learning to up-weight queries that successfully produce relevant results, we introduce $\underline{Le}$arning to $\underline{Re}$trieve by $\underline{T}$rying (LeReT), a reinforcement learning framework that explores search queries and uses preference-based optimization to improve their quality. LeReT can improve the absolute retrieval accuracy by up to 29% and the downstream generator evaluations by 17%. The simplicity and flexibility of LeReT allows it to be applied to arbitrary off-the-shelf retrievers and makes it a promising technique for improving general LLM pipelines. Project website: http://sherylhsu.com/LeReT/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。