让智能体学会判断每一步操作对最终结果的贡献,提升长任务决策能力。
PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment

- 用贝叶斯方法将最终结果反推到每一步操作的贡献度
- 在多轮搜索任务中显著提升成功轨迹的准确性与泛化性
- 适合需要长期推理和复杂决策的AI助手场景
长周期智能体任务中,基于结果的强化学习面临根本性的信用分配挑战:轨迹级奖励仅验证最终正确性,难以指导哪些中间推理步骤或工具调用真正促成结果。该问题在多轮搜索智能体中尤为突出,成功轨迹可能包含误导性操作,失败轨迹也可能包含有价值的证据收集步骤。本文提出PBSD(特权贝叶斯自蒸馏),一种基于贝叶斯校准的自蒸馏方法,实现稀疏最终奖励下的细粒度信用分配。PBSD通过验证答案的后验概率与先验概率之比衡量轨迹质量,并利用贝叶斯定理将难以估计的答案侧比率转化为标准学生模型与条件教师模型之间的可计算似然比。对这一贝叶斯证据得分进行自回归分解,生成逐回合信号,识别每一步是否支持或削弱已验证结果。因此,PBSD提供了一种原则性强、优雅的重加权方案,将稀疏结果监督转化为贝叶斯校准的回合级信用信号,且完全兼容标准策略优化。实验表明,PBSD在域内与域外设置下均持续提升性能,有效实现从短上下文训练到长上下文推理的知识迁移,表明其细粒度信用分配机制有助于更有效的策略学习并提升泛化能力。
原文摘要 · Abstract (English)
Long-horizon agentic tasks pose a fundamental credit assignment challenge for outcome-base reinforcement learning: trajectory-level rewards verify final correctness but provide limited guidance on which intermediate reasoning steps or tool interactions contribute to the outcome. The difficulty is especially pronounced in multi-turn search agents, where successful trajectories may contain misleading actions and failed trajectories may contain valuable evidence-gathering steps. We propose PBSD (Privileged Bayesian Self-Distillation), a Bayes-calibrated self-distillation method for fine-grained credit assignment under sparse final rewards. PBSD measures trajectory quality through the posterior-to-prior probability ratio of the verified answer and applies Bayes' rule to convert this hard-to-estimate answer-side ratio into a tractable likelihood ratio between a standard student model and a privileged answer-conditioned teacher model. Autoregressive decomposition of this Bayesian evidence score yields turn-level signals that identify whether each intermediate turn supports or undermines the verified outcome. Consequently, PBSD provides a principled and elegant reweighting scheme that transforms sparse outcome supervision into Bayes-calibrated turn-level credit signals, while remaining fully compatible with standard policy optimization. Experiments demonstrate that PBSD consistently enhances performance across both in-domain and out-of-domain settings, and effectively transfers knowledge from short-context training to long-context inference, suggesting that its fine-grained credit assignment mechanism facilitates more effective policy learning and yields improved generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。