EasyOPD统一了大模型在线策略蒸馏,让不同方法共用一套框架。
EasyOPD: An Easy-to-use On-Policy Distillation Framework for Large Language Models

- 将蒸馏逻辑与执行框架分离,支持多种监督形式
- 在推理、代码生成等任务上表现稳定,保持原有性能特点
- 提供可运行配置和演示,适合研究者快速复现
传统语言模型蒸馏依赖静态教师生成数据,难以覆盖学生策略演化过程中遇到的状态。在线策略蒸馏(OPD)则通过学生生成的轨迹收集教师或评估器的监督信号。然而现有OPD方法在监督形式、分词器兼容性、教师访问方式和监督粒度上差异显著,导致实现碎片化,难以复现与扩展。我们提出基于verl(一个用于大语言模型的分布式强化学习框架)的EasyOPD框架,将用户配置、方法特定的监督逻辑与verl执行解耦。其模块通过扩展边界连接共享后端,支持损失构建、回放元数据处理、奖励计算、分词器对齐及教师侧计算。我们实现了三种典型OPD场景:跨分词器蒸馏、在线自蒸馏和逐步蒸馏。在推理、代码生成、科学知识和工具使用基准测试中,这些方法均能通过同一verl后端运行,同时保留各自的方法目标与任务相关性能特征。我们开源了EasyOPD,包含可运行的YAML配置、文档和可安装的演示包及视频。
原文摘要 · Abstract (English)
Conventional language-model distillation often relies on fixed teacher-generated data, which may not cover the states encountered by an evolving student policy. On-policy distillation (OPD) instead collects teacher or evaluator supervision on student-generated rollouts. However, existing OPD methods differ substantially in supervision form, tokenizer compatibility, teacher access, and supervision granularity, leading to fragmented implementations that are difficult to reproduce and extend. We present \textsc{EasyOPD}, an on-policy distillation framework built on verl, a distributed reinforcement-learning framework for large language models. \textsc{EasyOPD} separates user-side configuration, method-specific supervision logic, and verl-based execution. Its method modules connect to the shared backend through extension boundaries for loss construction, rollout metadata, reward processing, tokenizer alignment, and teacher-side computation. We instantiate representative methods for three OPD settings -- cross-tokenizer OPD, on-policy self-distillation, and step-wise OPD. Experiments on reasoning, code-generation, scientific-knowledge, and tool-use benchmarks show that these implementations can be executed through the same verl-based backend while retaining their method-specific objectives and task-dependent performance profiles. We release \textsc{EasyOPD} with runnable YAML configurations, documentation, and an installable demonstration package and video.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。