通过答案回溯实现精准奖励，让长时序搜索智能体学会区分有用与冗余动作。
ABSeeker: Training Long-Horizon Search Agents via Answer-Backtracked Credit Assignment
- 从答案反推中间线索，将稀疏结果转化为每一步的精细奖励
- 仅用8.5k样本训练，4B模型在双数据集上达37.3%和39.1%准确率
- 适合需要多步推理的复杂任务，如信息检索与跨文档整合
长时序搜索智能体需执行多步操作以搜索、检索、验证并整合证据来得出最终答案。然而，现有训练方法在监督微调（SFT）和强化学习（RL）中对轨迹内所有步骤一视同仁，无法区分有效动作与错误或冗余动作。本文提出答案回溯信用分配（ABC），一种细粒度信用分配框架，通过将稀疏的轨迹级结果转化为密集的步骤级监督，奖励有用动作（即使在失败轨迹中）并抑制错误或冗余动作。具体而言，给定一个模糊查询及其真实答案，ABC首先进行答案回溯线索恢复，从答案反向追踪所需中间线索；然后采用线索锚定步骤评分，评估每一步是否符合这些线索，将稀疏的二值结果监督转化为密集的步骤级奖励。基于此奖励，我们构建ABC-SFT，重加权每轮损失；以及ABC-GRPO，将步骤级得分作为GRPO中的奖励。在此框架下，我们基于Qwen3.5-4B仅用8.5k样本训练出ABSeeker，其在BrowseComp上达37.3%，在BrowseComp-ZH上达39.1%。启用上下文管理后，性能进一步提升至55.3%和52.9%，显著优于同规模（4B）模型，甚至媲美约30B的大模型。结果证明，答案回溯的步骤级信用分配在训练长时序搜索智能体方面极具有效性。
原文摘要 · Abstract (English)
Long-horizon search agents must make multiple sequential actions (steps) to search, retrieve, verify, and integrate evidence to reach a final answer. However, existing methods for training these agents typically treat all steps within a trajectory uniformly during both supervised fine-tuning (SFT) and reinforcement learning (RL), failing to distinguish useful actions from erroneous or redundant ones. In this paper, we propose Answer-Backtracked Credit Assignment (ABC), a fine-grained credit assignment framework for training long-horizon search agents by converting sparse trajectory-level outcomes into dense step-level supervision that rewards useful actions (even in failed trajectories) while suppressing erroneous or redundant actions. Specifically, given a potentially obscure query and its corresponding ground-truth answer, ABC first performs Answer-Backtracked Clue Recovery, which traces back from the answer to recover intermediate clues required to solve the question. It then applies Clue-Anchored Step Scoring to evaluate each search step against these clues, converting sparse binary outcome supervision into dense step-level rewards. Based on these rewards, we develop ABC-SFT, which reweights the loss of each turn, and ABC-GRPO, which uses the step-level scores as rewards in GRPO. Building on this framework, we train ABSeeker based on Qwen3.5-4B with only 8.5k examples. ABSeeker achieves 37.3% on BrowseComp and 39.1% on BrowseComp-ZH. With context management, the scores further improve to 55.3% and 52.9%, respectively, significantly outperforming same-scale (4B) agents and even matching the performance of larger ones (approximately 30B). These results demonstrate the effectiveness of answer-backtracked step-level credit assignment for training long-horizon search agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。