让大模型智能体在长任务中更稳定地学习，提升成功率。
Self-Distilled Agentic Reinforcement Learning
- 用教师分支提供细粒度指导，结合强化学习主干优化
- 在ALFWorld等任务上比GRPO提升最高9.4%以上
- 适合需要长期决策的智能体训练，尤其大模型场景
强化学习（RL）是训练大语言模型智能体的核心范式，但其轨迹级奖励信号对长程交互仅提供粗粒度监督。自蒸馏方法（OPSD）通过引入带特权上下文的教师分支，提供密集的词元级指导，但将其应用于多轮智能体时面临挑战：多轮不稳定性会加剧监督失效，且技能条件下的特权指导需对负面教师拒绝进行不对称处理，可能因技能检索或使用不完善导致。本文提出SDAR（Self-Distilled Agentic Reinforcement Learning），将OPSD作为门控辅助目标，保持RL为优化主干。SDAR通过Sigmoid门控将分离的词元级信号映射，增强教师认可的正向差距词元的蒸馏强度，同时软性抑制负向教师拒绝。在Qwen2.5与Qwen3系列模型上，于ALFWorld、WebShop和Search-QA数据集测试中，SDAR显著优于GRPO（ALFWorld+9.4%，Search-QA+7.0%，WebShop-Acc+10.2%），避免了直接组合GRPO+OPSD的不稳定性，并在不同模型规模下持续超越混合式基线。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has emerged as a central paradigm for post-training LLM agents, yet its trajectory-level reward signal provides only coarse supervision for long-horizon interaction. On-Policy Self-Distillation (OPSD) complements RL by introducing dense token-level guidance from a teacher branch augmented with privileged context. However, transferring OPSD to multi-turn agents proves problematic: compounding multi-turn instability destabilizes supervision, while skill-conditioned privileged guidance requires asymmetric treatment for negative teacher rejections may arise from imperfect skills retrieval or utilization. We introduce SDAR (Self-Distilled Agentic Reinforcement Learning), which treats OPSD as a gated auxiliary objective while keeping RL as the primary optimization backbone. SDAR maps detached token-level signals into a sigmoid gate, strengthening distillation on teacher-endorsed positive-gap tokens and softly attenuating negative teacher rejections. Across the Qwen2.5 and Qwen3 families on ALFWorld, WebShop, and Search-QA, SDAR substantially improves over GRPO (+9.4% on ALFWorld, +7.0% on Search-QA, +10.2% on WebShop-Acc), avoids the instability of naive GRPO+OPSD, and consistently outperforms hybrid RL--OPSD baselines across model scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。