用大模型指导小模型解数学题，准确率提升近5个百分点。
Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition
- 用双代理框架构建思维链数据集，小模型通过低秩微调学习。
- 小模型准确率达69.43%，比基线高4.76个百分点，跨数据集泛化至73.1%。
- 答案越短准确率越低，尤其在快速答题环节敏感度最高。
本文研究将大型推理模型 DeepSeek-R1 的知识蒸馏到紧凑的学生模型 Qwen2.5-7B。基于北肯塔基大学约翰·奥布莱恩数学竞赛（2011–2025）的历史题目，通过双代理框架构建思维链（CoT）训练语料。在 Apple Silicon 硬件上使用 MLX 框架，采用低秩适配（LoRA）对学生模型进行微调。基础 Qwen2.5-7B 模型在竞赛题上准确率为 64.67%，而教师模型 DeepSeek-R1 达到 91.40%。初始 1,000 次迭代训练显示严重过拟合，验证损失在第 200 次迭代后持续上升。据此，我们进行五次独立训练，每轮限 200 次迭代，随机种子不同。五次运行平均准确率为 69.43%（标准差 0.17%），较基线提升 4.76 个百分点；在 MATH-500 基准上泛化准确率达 73.1%（标准差 0.18%）。进一步分析发现，回答长度影响推理质量：从 R1（平均 220 字）到 R6（平均 31.2 字），准确率由 69.43% 降至 41.9%，其中两人速答环节最易受词数减少影响。结果表明，思维链蒸馏有效提升小模型性能，且回答长度是数学推理质量的关键因素。
原文摘要 · Abstract (English)
This paper investigates knowledge distillation from a large reasoning model (DeepSeek-R1) to a compact student model (Qwen2.5-7B). Using historical problems from the John O'Bryan Mathematics Competition at Northern Kentucky University (2011-2025), we build a Chain-of-Thought (CoT) training corpus through a dual-agent framework. The dataset is used to fine-tune the student model with Low-Rank Adaptation (LoRA) on Apple Silicon hardware using the MLX framework. The base Qwen2.5-7B model achieves 64.67% accuracy on competition problems, while the DeepSeek-R1 teacher achieves 91.40%. An initial 1,000-iteration training run revealed severe overfitting, with validation loss reaching a minimum at iteration 200 before rising steadily. Based on this finding, we ran five independent training runs each limited to 200 iterations with varied random seeds to assess result stability. Across these five runs, the fine-tuned student model achieves a mean accuracy of 69.43% (std dev 0.17%) on the competition dataset, a 4.76 percentage-point improvement over the base model, and generalizes to 73.1% (std dev 0.18%) on the MATH-500 benchmark. We further study how response length affects answer quality across six reasoning levels (R1-R6): accuracy declines consistently from 69.43% at R1 (mean 220 words) to 41.9% at R6 (mean 31.2 words), with the two-person speed section most sensitive to token reduction. These results demonstrate that CoT distillation improves compact student models and that response length is a critical factor in mathematical reasoning quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。