用极少数据训练搜索代理，提升大模型问答准确率
s3: You Don't Need That Much Data to Train a Search Agent via RL
- 分离检索与生成模块，仅优化搜索器
- 仅需2400样本即超越超大规模数据训练的基线
- 适合想低成本提升检索能力的开发者使用
检索增强生成（RAG）系统使大语言模型（LLMs）能在推理时访问外部知识。近期进展已实现通过强化学习（RL）让LLMs充当搜索代理，通过与检索引擎的多轮交互提升信息获取能力。然而，现有方法要么仅优化检索指标（如NDCG），忽略下游实用价值；要么微调整个LLM，导致检索与生成耦合，限制了实际搜索效果及对冻结或专有模型的兼容性。本文提出s3，一种轻量级、模型无关的框架，将搜索器与生成器解耦，并采用‘超越RAG的收益’作为奖励信号——即生成准确率相较于朴素RAG的提升。s3仅需2.4k训练样本，便在六项通用问答和五项医学问答基准上持续优于使用超过70倍数据训练的基线模型。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) systems empower large language models (LLMs) to access external knowledge during inference. Recent advances have enabled LLMs to act as search agents via reinforcement learning (RL), improving information acquisition through multi-turn interactions with retrieval engines. However, existing approaches either optimize retrieval using search-only metrics (e.g., NDCG) that ignore downstream utility or fine-tune the entire LLM to jointly reason and retrieve-entangling retrieval with generation and limiting the real search utility and compatibility with frozen or proprietary models. In this work, we propose s3, a lightweight, model-agnostic framework that decouples the searcher from the generator and trains the searcher using a Gain Beyond RAG reward: the improvement in generation accuracy over naive RAG. s3 requires only 2.4k training samples to outperform baselines trained on over 70x more data, consistently delivering stronger downstream performance across six general QA and five medical QA benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。