让小模型通过重放前缀和在线续写，更高效学习大模型的多轮决策。
From Trajectories to Prefixes: Reusing Teacher Trajectories via Replayed Prefixes and Online Continuation
- 将教师轨迹拆分为可重放的前缀和在线续写部分，分步训练。
- 在三个环境上提升小模型性能，较蒸馏方法显著改进。
- 适合想用小模型实现复杂交互任务的研究者与开发者。
小型语言模型是交互式智能体的理想骨干，但直接从强教师轨迹中蒸馏会将丰富的多轮行为简化为单次模仿目标，效率低下。针对长时程环境中的早期决策影响后续状态与奖励的问题，我们提出 Prefix-GRPO：一种强化学习框架，将教师轨迹分解为可重放的前缀查询与在线续写。每个前缀在环境中重放以恢复有效中间状态，随后学生模型在线继续交互并获得任务奖励。与仅更新响应的 GRPO 不同，Prefix-GRPO 还对重放前缀中的历史助手标记应用裁剪策略更新，使用策略蒸馏的 SFT 检查点估计其旧对数概率。该方法统一了前缀学习与续写学习的优化形式。在 TextCraft、BabyAI 与 ALFWorld 上的实验表明，Prefix-GRPO 显著优于蒸馏与标准强化学习基线；消融实验显示，仅重放不足以取得效果，必须显式优化前缀标记。代码与复现脚本已开源。
原文摘要 · Abstract (English)
Small language models are attractive backbones for interactive agents, but direct distillation from strong teacher trajectories often turns rich multi-turn behavior into one-shot imitation targets. This is inefficient in long-horizon environments, where early decisions shape later states and rewards. We propose Prefix-GRPO, a reinforcement learning framework that decomposes teacher trajectories into replay-aligned prefix queries and online continuations. Each prefix is replayed in the environment to recover a valid intermediate state, after which the student continues online interaction and receives task reward. Unlike response-only GRPO, Prefix-GRPO also applies clipped policy updates to historical assistant tokens inside the replayed prefix, using a policy-distilled SFT checkpoint to estimate their old log-probabilities. This unifies prefix learning and continuation learning within the same policy-optimization form. Experiments on TextCraft, BabyAI, and ALFWorld show that Prefix-GRPO improves small-model agents over distillation and standard RL baselines, while ablations show that replay alone is insufficient without explicit prefix-token optimization. The implementation and reproduction scripts are available at https://github.com/HappynessI/Prefix_GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。