用回溯批评机制让搜索智能体每一步都得到精准反馈，训练更稳定高效。
CriticSearch: Fine-Grained Credit Assignment for Search Agents via a Retrospective Critic
- 通过回溯式批评模型逐轮评估，提供密集精细的奖励信号。
- 在多跳推理任务上收敛更快，性能超越现有基线模型。
- 适合需要稳定训练和高精度推理的复杂问答场景。
集成工具的推理（TIR）使大语言模型能迭代调用搜索引擎获取最新外部知识，提升复杂问答任务中的适应性与泛化能力。然而，现有搜索代理管道多依赖强化学习优化，常因结果奖励稀疏导致探索效率低、训练不稳定。本文提出CriticSearch，一种细粒度信用分配框架，通过回溯批评机制提供逐轮的密集反馈。训练时，一个冻结的非对称批评大模型利用完整轨迹和标准答案的特权信息，逐轮评估并生成稳定、密集的奖励信号，指导策略改进。在多个多跳推理基准上的实验表明，CriticSearch持续优于现有基线，实现更快收敛、更高训练稳定性与更强性能。
原文摘要 · Abstract (English)
Tool-Integrated Reasoning (TIR) with search engines enables large language models to iteratively retrieve up-to-date external knowledge, enhancing adaptability and generalization in complex question-answering tasks. However, existing search agent pipelines typically depend on reinforcement learning based optimization, which often suffers from sparse outcome rewards, leading to inefficient exploration and unstable training. We introduce CriticSearch, a fine-grained credit-assignment framework that supplies dense, turn-level feedback via a retrospective critic mechanism. During training, a frozen, asymmetric critique LLM retrospectively evaluates each turn using privileged information from the full trajectory and gold answers, converting these assessments into stable, dense rewards that guide policy improvement. Experimental results across diverse multi-hop reasoning benchmarks demonstrate that CriticSearch consistently outperforms existing baselines, achieving faster convergence, improved training stability, and higher performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。