arXiv:2505.07903cs.CLcs.AI2025-05被引 9

让大模型学会何时搜、何时不搜,提升推理效率。

SEM: Reinforcement Learning for Search-Efficient Large Language Models

  • 用强化学习训练模型区分该用内知还是外搜。
  • 减少冗余搜索,准确率反而提升或保持不变。
  • 适合想优化大模型检索策略的研究者和开发者。

近年来,大语言模型(LLMs)不仅在推理方面表现出色,还能调用外部工具(尤其是搜索引擎)。然而,如何让模型判断何时调用搜索、何时依赖内部知识仍是一大挑战。现有强化学习方法常导致重复搜索,造成效率低下和成本过高。本文提出SEM——一种新型的后训练强化学习框架,旨在显式优化大模型的搜索使用。通过构建融合MuSiQue与MMLU的平衡数据集,设计结构化推理模板,并采用组相对策略优化(GRPO)对模型进行后训练。奖励函数鼓励在无需搜索时避免调用,同时在需要时有效检索。实验表明,该方法显著减少了冗余搜索,且在多个高难度基准上维持甚至提升了答案准确率。该框架增强了模型的推理效率,扩展了其合理利用外部知识的能力。

原文摘要 · Abstract (English)

Recent advancements in Large Language Models(LLMs) have demonstrated their capabilities not only in reasoning but also in invoking external tools, particularly search engines. However, teaching models to discern when to invoke search and when to rely on their internal knowledge remains a significant challenge. Existing reinforcement learning approaches often lead to redundant search behaviors, resulting in inefficiencies and over-cost. In this paper, we propose SEM, a novel post-training reinforcement learning framework that explicitly trains LLMs to optimize search usage. By constructing a balanced dataset combining MuSiQue and MMLU, we create scenarios where the model must learn to distinguish between questions it can answer directly and those requiring external retrieval. We design a structured reasoning template and employ Group Relative Policy Optimization(GRPO) to post-train the model's search behaviors. Our reward function encourages accurate answering without unnecessary search while promoting effective retrieval when needed. Experimental results demonstrate that our method significantly reduces redundant search operations while maintaining or improving answer accuracy across multiple challenging benchmarks. This framework advances the model's reasoning efficiency and extends its capability to judiciously leverage external knowledge.

大模型强化学习搜索优化推理效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。