用小模型专攻搜索规划,让大模型更高效答题。
AI-SearchPlanner: Modular Agentic Search via Pareto-Optimal Multi-Objective Reinforcement Learning
- 拆分搜索规划与生成模块,小模型专注优化搜索路径。
- 双奖励对齐机制提升规划准确率,实验显示性能超越现有方法。
- 适配各类大模型和数据领域,通用性强,适合部署复杂搜索系统。
近期研究尝试将大语言模型(LLMs)与搜索引擎结合,利用模型内嵌知识与外部信息。强化学习(RL)已成为通过多轮交互提升LLM推理能力的有力范式。然而,现有基于RL的搜索代理通常由单一LLM端到端完成搜索规划与问答任务,难以同时优化两者。实际中,高性能AI搜索系统常使用大而冻结的LLM(如GPT-4、DeepSeek-R1)保障问答质量。因此,更优策略是使用小型可训练LLM专注搜索规划。本文提出新型强化学习框架AI-SearchPlanner,聚焦于增强冻结问答模型性能。核心创新包括:1)解耦搜索规划器与生成器架构;2)双奖励对齐机制;3)帕累托优化规划效用与成本。在真实数据集上的大量实验表明,该方法在有效性与效率上均优于现有基于RL的搜索代理,并展现出跨不同冻结问答模型与数据领域的强泛化能力。
原文摘要 · Abstract (English)
Recent studies have explored integrating Large Language Models (LLMs) with search engines to leverage both the LLMs' internal pre-trained knowledge and external information. Specially, reinforcement learning (RL) has emerged as a promising paradigm for enhancing LLM reasoning through multi-turn interactions with search engines. However, existing RL-based search agents rely on a single LLM to handle both search planning and question-answering (QA) tasks in an end-to-end manner, which limits their ability to optimize both capabilities simultaneously. In practice, sophisticated AI search systems often employ a large, frozen LLM (e.g., GPT-4, DeepSeek-R1) to ensure high-quality QA. Thus, a more effective and efficient approach is to utilize a small, trainable LLM dedicated to search planning. In this paper, we propose \textbf{AI-SearchPlanner}, a novel reinforcement learning framework designed to enhance the performance of frozen QA models by focusing on search planning. Specifically, our approach introduces three key innovations: 1) Decoupling the Architecture of the Search Planner and Generator, 2) Dual-Reward Alignment for Search Planning, and 3) Pareto Optimization of Planning Utility and Cost, to achieve the objectives. Extensive experiments on real-world datasets demonstrate that AI SearchPlanner outperforms existing RL-based search agents in both effectiveness and efficiency, while exhibiting strong generalization capabilities across diverse frozen QA models and data domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。