用离散评分替代逐词对齐，实现高效在线蒸馏。
OVD: On-policy Verbal Distillation
- 用0-9离散评分替代逐词概率匹配，降低对齐要求。
- 在网页问答任务上提升12.9%准确率，数学推理提升25.7%。
- 适合需要高效训练与自由探索的强化学习场景。
知识蒸馏为将大模型的推理能力迁移到高效小模型提供了可行路径；然而，现有的逐标记在线蒸馏方法需在学生与教师模型间进行逐标记对齐，限制了学生模型的探索能力，无法有效利用交互环境反馈，并在强化学习中面临严重内存瓶颈。我们提出一种名为「在线话语蒸馏」（OVD）的内存高效框架，用教师模型输出的离散话语评分（0–9）进行轨迹匹配，替代逐标记概率匹配。OVD 显著降低内存消耗，支持基于话语反馈的在线蒸馏，避免逐标记对齐，使学生模型可自由探索输出空间。在网页问答和数学推理任务上的大量实验表明，OVD 显著优于现有方法，在网页问答任务上平均精确匹配（EM）最高提升12.9%，在数学基准测试中提升达25.7%（仅使用一次随机样本训练），同时具备更优训练效率。
原文摘要 · Abstract (English)
Knowledge distillation offers a promising path to transfer reasoning capabilities from large teacher models to efficient student models; however, existing token-level on-policy distillation methods require token-level alignment between the student and teacher models, which restricts the student model's exploration ability, prevent effective use of interactive environment feedback, and suffer from severe memory bottlenecks in reinforcement learning. We introduce On-policy Verbal Distillation (OVD), a memory-efficient framework that replaces token-level probability matching with trajectory matching using discrete verbal scores (0--9) from teacher models. OVD dramatically reduces memory consumption while enabling on-policy distillation from teacher models with verbal feedback, and avoids token-level alignment, allowing the student model to freely explore the output space. Extensive experiments on Web question answering and mathematical reasoning tasks show that OVD substantially outperforms existing methods, delivering up to +12.9% absolute improvement in average EM on Web Q&A tasks and a up to +25.7% gain on math benchmarks (when trained with only one random samples), while also exhibiting superior training efficiency. Our project page is available at https://OVD.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。