让大模型通过事后分析失败轨迹，提升搜索强化学习效果。
HindSearch: Trajectory-Level Hindsight Critique for Search-Augmented Reinforcement Learning
- 用事后批判机制分析失败轨迹，生成辅助训练信号。
- 在7个基准上平均精确匹配达39.4%，超越现有方法。
- 适合想提升推理与搜索能力的RL系统研究者。
搜索增强型语言模型代理通常使用二元精确匹配奖励，忽略了失败轨迹中大量有价值的信息。本文提出HindSearch，一种针对GRPO的后见式自蒸馏方法：每轮推演后，一个冻结的评判器利用标准答案对每个失败轨迹撰写简短批判，该批判作为辅助的在线策略蒸馏信号，用于优化学生模型的搜索行为。在包含7个基准的标准评测集上，基于Qwen2.5-3B-Instruct的HindSearch达到39.4%的平均精确匹配率，显著优于先前的搜索强化学习基线。当评判器无法访问标准答案时，性能提升几乎消失，表明改进主要源于后见式分析机制。
原文摘要 · Abstract (English)
Search-augmented LM agents are typically trained with a binary exact-match reward, which throws away most of what a failed trajectory tells us about why it failed. We introduce HindSearch, a hindsight self-distillation procedure for GRPO: after each rollout, a frozen judge writes a short critique of every failed trajectory using the gold answer, and the critique supplies an auxiliary on-policy distillation signal on the student's search actions. On the standard seven-benchmark suite with Qwen2.5-3B-Instruct, HindSearch reaches 39.4% average EM, outperforming prior search-RL baselines. Removing the judge's access to the gold answer erases most of the gain, isolating hindsight as the source of the improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。