让智能体自己生成教学信号,实现更高效的自我进化。
Latent On-Policy Self-Distillation

- 用可学习的隐变量作为教师的教学信号,替代人工设计的监督信息。
- 在工具使用和代码生成任务中超越现有方法,且训练开销低于30%。
- 适合追求自主进化与高效学习的智能体研究者。
让智能体从经验中学习并内化为策略,已成为自演化AI的核心问题。当前的在线策略自蒸馏(OPSD)方法依赖于人工设计的特权信息(如答案、反馈、技能或轨迹),限制了端到端可学习性和持续自我改进的可扩展性。本文提出隐式在线策略自蒸馏(LOPD),将教师的特权上下文本身变为可端到端学习的隐变量。LOPD通过检索相关经验并组合成连续隐变量令牌来构建自教师,学生则根据任务与交互历史生成轨迹,并在每个访问前缀处接收细粒度的令牌级监督。我们进一步引入特权边界目标以稳定隐变量的学习。实验表明:(I)LOPD性能优异,在代理工具使用与代码生成任务中优于RLVR及主流OPSD方法(OPSD、SDPO、Skill-SD);(II)学习效率高,仅需不到30%的回放预算即超越GRPO与Skill-SD。消融实验进一步证明,使特权上下文可学习是获得这些提升的关键。结果表明,LOPD推动了更具可扩展性与自导向的智能体演化范式。
原文摘要 · Abstract (English)
Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher's privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。