用可学习的内在奖励,让搜索型智能体更智能地决定何时该搜、是否真用了证据。
Search-G1: Grounded Search Agents via Representation-Based Intrinsic Rewards

- 通过两个读出层评估答案是否依赖证据、是否需要搜索
- 在多个基准上实现更少搜索次数,同时保持高准确率
- 无需人工标注或大模型打分,适合部署到资源受限场景
搜索增强型语言智能体应仅在必要时检索外部信息,并基于检索证据回答问题。现有外部奖励要么反馈稀疏(仅结果正确性),要么依赖昂贵的过程标注或大模型评判。基于策略侧信号(如熵、似然、信息增益)的内在奖励虽易计算,但主要反映模型自信度而非证据依赖性。本文提出 Search-G1,一种基于表示的内在奖励框架,通过两个校准后的读出层衡量智能体答案的操作性接地性:提示状态读出预测闭卷知识是否足够,其补集定义了策略相对的检索必要性;答案提交读出估计答案对证据删除的敏感度,反映对证据的依赖程度。两者共同为必要且依赖证据的搜索路径提供额外奖励,鼓励闭卷即可作答的情况,惩罚重复搜索。训练中无需过程标注或大模型评判,奖励可随策略更新周期性重校准。在多个搜索问答基准及两种模型规模下实验表明,Search-G1在保持任务准确率的同时,显著提升接地性与搜索成本之间的平衡,生成更短的回答轨迹。代码已开源。
原文摘要 · Abstract (English)
Search-augmented language agents should retrieve external information only when necessary and ground their answers in retrieved evidence. Existing external rewards provide either sparse outcome supervision or richer feedback from process annotations and LLM judges. Outcome rewards scale readily but cannot distinguish grounded retrieval from redundant search, whereas richer signals require costly annotation or inference during training. Internal rewards based on policy-side signals such as entropy, likelihood, or information gain are graded and inexpensive to evaluate, yet mainly reflect model confidence rather than evidence grounding. We propose Search-G1, a representation-based intrinsic reward framework that measures the operational grounding of an agent's answers through two intervention-calibrated readouts. A prompt-state readout predicts closed-book sufficiency, whose complement defines policy-relative retrieval necessity; an answer-commit readout estimates evidence reliance from answer-stage sensitivity to evidence deletion. Together, they provide additional credit to correct searched trajectories when retrieval is estimated necessary and the answer is evidence-sensitive, favor correct direct answers when closed-book knowledge suffices, and penalize repeated search. After calibration, reward scoring requires neither process annotations nor LLM-as-judge inference during policy optimization. Because reinforcement learning changes policy representations, Search-G1 periodically refits both readouts on trajectories from the latest checkpoint, allowing the reward to co-evolve with the policy. Experiments across multiple search-based question-answering benchmarks and two model scales show that Search-G1 improves the grounding--search-cost trade-off, producing shorter response-side trajectories at competitive task accuracy. Code is available at https://github.com/Rosy0912/Search-G1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。