让AI agent自己反思成功与失败轨迹，提升决策精准度。
Group-Reflective Self-Distillation for Agentic Reinforcement Learning
- 用自身验证过的轨迹分组反思，生成适配当前能力的指导信号。
- 通过对比成功与失败轨迹的反思，优化每一步的奖励分配。
- 适用于大模型智能体训练，尤其在新任务上泛化能力更强。
基于可验证奖励的强化学习（RLVR）在训练大型语言模型智能体方面表现有效。然而，终端奖励仅提供粗粒度的轨迹级监督，导致成功行为、重复错误和偶然选择混杂在同一结果信号中。现有智能体自蒸馏方法虽通过自然语言技能增强稀疏监督，但外部检索或由更强模型从单条轨迹提取的技能可能与当前经验不匹配、超出策略能力范围，或仅具路径特异性。本文提出群组反射式自蒸馏（GRSD），从策略自身的验证回放中提取对齐能力且能区分结果的指导信息。针对每个提示，策略在同策略群组内反思每条验证轨迹，通过停止梯度快照对比成功与失败回放的反思结果，构建群体级优势指导。在此指导下，自教师通过调节基于结果的优势来优化回合级信用分配，同时保持验证器决定的学习方向。在多个智能体环境和不同模型规模下的实验表明，GRSD持续优于竞争性基线，并在未见任务上展现出更强泛化能力。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) is effective for training large language model agents. However, terminal rewards provide only coarse trajectory-level supervision, leaving successful behaviors, recurring mistakes, and incidental choices entangled in the same outcome signal. Existing agentic self-distillation methods enrich sparse supervision with natural-language skills, but skills retrieved externally or extracted from a single trajectory by stronger models may mismatch current experience, exceed the policy's capability, or remain path-specific. We propose Group-Reflective Self-Distillation (GRSD), which derives capability-aligned and outcome-discriminative guidance from the policy's own verified rollouts. For each prompt, the policy reflects on each verified trajectory in an on-policy group, and a stop-gradient snapshot contrasts the resulting reflections from successful and failed rollouts to construct group-level privileged guidance. Conditioned on this guidance, a self-teacher refines turn-level credit assignment by modulating outcome-based advantages while preserving the verifier-determined learning direction. Experiments across multiple agentic environments and model scales demonstrate that GRSD consistently outperforms competitive baselines and generalizes more effectively to unseen tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。