让语言模型从自身经验中学习并内化知识，提升推理能力。
On-Policy Context Distillation for Language Models
- 用自生成轨迹训练学生模型，最小化与教师模型的反KL散度
- 在数学推理等任务上准确率更高，且保持对新场景的适应力
- 适合想提升模型泛化能力的研究者和工程师
上下文蒸馏使语言模型能够将上下文中的知识内化到参数中。本文提出一种名为在线策略上下文蒸馏（OPCD）的框架，通过在学生模型自生成的轨迹上训练，并最小化其与上下文条件教师模型之间的反向Kullback-Leibler散度，将在线策略蒸馏与上下文蒸馏相结合。我们在两个重要应用中验证了OPCD的有效性：经验知识蒸馏，即模型从自身历史求解轨迹中提取并整合可迁移知识；系统提示蒸馏，即模型内化优化提示中编码的有益行为。在数学推理、文本游戏和领域特定任务中，OPCD consistently优于基线方法，在提高任务准确率的同时更好保留了分布外能力。此外，我们证明OPCD可实现跨规模蒸馏，使小模型从大模型中内化经验知识。
原文摘要 · Abstract (English)
Context distillation enables language models to internalize in-context knowledge into their parameters. In our work, we propose On-Policy Context Distillation (OPCD), a framework that bridges on-policy distillation with context distillation by training a student model on its own generated trajectories while minimizing reverse Kullback-Leibler divergence against a context-conditioned teacher. We demonstrate the effectiveness of OPCD on two important applications: experiential knowledge distillation, where models extract and consolidate transferable knowledge from their historical solution traces, and system prompt distillation, where models internalize beneficial behaviors encoded in optimized prompts. Across mathematical reasoning, text-based games, and domain-specific tasks, OPCD consistently outperforms baseline methods, achieving higher task accuracy while better preserving out-of-distribution capabilities. We further show that OPCD enables effective cross-size distillation, where smaller student models can internalize experiential knowledge from larger teachers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。