用协作生成法提升推理模型微调效果，避免性能下降。
How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data
- 师生模型交替生成文本，融合教师能力与学生风格
- 在代码生成任务中，性能提升最高达11.25%
- 适合需要稳定微调推理模型的研究者
主流模型增强策略是使用更强模型生成合成数据进行监督微调（SFT）。然而，对于新兴推理模型如Qwen3-8B，该方法常无法提升推理能力，甚至导致性能显著下降。本文发现，教师生成数据与学生分布之间存在显著风格差异是关键问题。为此，提出教师-学生协作数据合成框架TESSY，通过交替生成风格与非风格标记，使合成数据既继承教师的先进推理能力，又保持与学生一致的风格分布。实验以GPT-OSS-120B为教师，在代码生成任务上，对Qwen3-8B使用教师生成数据微调，导致LiveCodeBench-Pro性能下降3.25%，OJBench下降10.02%；而TESSY实现11.25%和6.68%的提升。
原文摘要 · Abstract (English)
A widely adopted strategy for model enhancement is to use synthetic data generated by a stronger model for supervised fine-tuning (SFT). However, for emerging reasoning models like Qwen3-8B, this approach often fails to improve reasoning capabilities and can even lead to a substantial drop in performance. In this work, we identify substantial stylistic divergence between teacher generated data and the distribution of student as a major factor impacting SFT. To bridge this gap, we propose a Teacher-Student Cooperation Data Synthesis framework (TESSY), which interleaves teacher and student models to alternately generate style and non-style tokens. Consequently, TESSY produces synthetic sequences that inherit the advanced reasoning capabilities of the teacher while maintaining stylistic consistency with the distribution of the student. In experiments on code generation using GPT-OSS-120B as the teacher, fine-tuning Qwen3-8B on teacher-generated data leads to performance drops of 3.25% on LiveCodeBench-Pro and 10.02% on OJBench, whereas TESSY achieves improvements of 11.25% and 6.68%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。