arXiv:2608.14277cs.CLcs.AI2026-08

用简单方法让短模型学会长文本推理,效果显著提升。

SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning

论文配图:SimpleOPD: Simple Tokenizer-Agnostic On-Policy Distillation for Long-Context Reasoning
图 1 · 摘自论文原文
  • 在共享文本空间对齐相同文本片段,解决分词器不一致问题。
  • 引入参考KL损失控制生成长度,使学生模型稳定增长。
  • 适用于多类模型,尤其在数学证明任务上提升明显。

基于策略的蒸馏(OPD)为将强教师模型的推理能力迁移到较弱学生模型提供了新路径,但在长上下文教师与短上下文学生之间应用时面临分词器差异、师生分布不匹配、输出长度爆炸和训练不稳定等挑战。本文以长上下文推理模型SU-01为教师,向Qwen3、Qwen3.5、Intern-S2、GLM-4.7、Gemma-4等短上下文学生模型迁移证明推理能力。通过在共享文本空间中对齐相同文本跨度的标记,缓解分词器差异;引入学生参考KL损失并屏蔽特殊终止标记如</think>和<|im_end|>的优势,抑制生成过长和频繁截断,从而缓解分布偏移,实现稳定的长度增长。实验显示,在同族与异族学生模型上均取得一致提升,尤其在自然语言数学证明任务上表现突出:Intern-S2-Preview在ProofBench上提升21.2点至55.2,超过Gemini-2.5-Pro;同时在科学基准HLE和HiPhO上也有改善,表明该方法可迁移泛化于数学以外领域。

原文摘要 · Abstract (English)

On-policy distillation (OPD) offers a promising way to transfer reasoning capabilities from stronger teacher models, but applying it to long-context reasoning teachers and short-context students introduces practical challenges, including tokenizer mismatch, teacher-student distribution mismatch, response length explosion, and training instability. In this work, we study this setting by transferring proof-reasoning capabilities from the long-context reasoning model SU-01 to short-context student models. To handle tokenizer differences, we perform OPD in a shared text space and align only tokens that occupy identical text spans under the student and teacher tokenizers. To mitigate the problem of excessive generation length and frequent truncation, we introduce a student reference KL loss and mask the advantages of special termination tokens such as </think> and <|im_end|>. This strategy constrains the student from drifting excessively from its initial policy, thereby mitigating the teacher-student distribution mismatch problem and fostering steady length growth. Experiments on both same-family and different-family student models, including Qwen3, Qwen3.5, Intern-S2, GLM-4.7, Gemma-4, show consistent gains in mathematical reasoning, especially natural-language math proving. Notably, Intern-S2-Preview improves by 21.2 points on ProofBench, reaching 55.2 and surpassing Gemini-2.5-Pro. It also improves on science benchmarks such as HLE and HiPhO, suggesting that OPD transfers reasoning capabilities that generalize beyond the mathematical training domain.

推理蒸馏长文本推理数学证明模型压缩

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。