让大模型学会何时该搜、何时该停,减少无效搜索
SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Search

- 通过对比有无搜索的推理路径,动态识别模型知识边界
- 引入惩罚机制,使搜索行为更精准,减少40%以上冗余搜索
- 适合追求高效推理的智能搜索系统开发者
代理式搜索使大语言模型通过迭代推理和外部搜索解决复杂多跳问题。然而实践中存在关键缺陷:模型无法识别自身知识边界,导致在已有知识足够时仍盲目触发搜索,或在已收集足够证据后仍持续搜索。这种缺乏自我意识的行为引发严重过搜索,带来显著推理延迟和高昂计算成本。为此,我们提出SAAS,一种新型强化学习框架,旨在培养动态自我意识,精确调控搜索行为而不影响准确性。SAAS包含三个核心组件:(i) 搜索边界建模机制,通过对比禁用搜索与启用搜索的轨迹,识别策略演化下的搜索边界;(ii) 边界感知奖励模块,将边界意识转化为轨迹级惩罚,抑制不必要的冗余搜索;(iii) 分阶段优化策略,采用序列课程训练优先强化推理能力,避免奖励劫持。大量实验表明,SAAS显著减少过搜索,同时保持高准确率。代码与实现细节已公开于https://github.com/XMUDeepLIT/SAAS。
原文摘要 · Abstract (English)
Agentic search enables LLMs to solve complex multi-hop questions through iterative reasoning and external search. Despite the effectiveness, these systems often suffer from a critical limitation in practice: agents fail to recognize their own knowledge boundaries, blindly triggering searches when internal knowledge suffices and failing to terminate search even when adequate evidence has been collected. The lack of self-awareness leads to severe \textbf{over-search}, incurring substantial inference latency and prohibitive computational cost. To this end, we propose SAAS, a novel RL framework designed to cultivate dynamic self-awareness that precisely regulates search behavior without compromising accuracy. SAAS introduces three key components: (i) a search boundary modeling mechanism, which identifies the search boundary under the evolving policy by contrasting search-disabled and search-enabled rollouts; (ii) a boundary-aware reward module, which translates this boundary awareness into trajectory-level penalties, suppressing unnecessary and redundant searches; and (iii) a stage-wise optimization strategy, which leverages a sequential curriculum to prioritize reasoning over search regularization, thereby avoiding reward hacking. Extensive experiments demonstrate that SAAS substantially reduces over-search, while maintaining accuracy. Our code and implementation details are released at https://github.com/XMUDeepLIT/SAAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。