发现代码推理蒸馏存在性能先降后升的特殊缩放规律
The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Models
- 通过增加训练数据量研究小模型代码推理能力的演化
- 性能先下降后以超对数速度上升，形成'代码推理谷地'
- 难易题目对学习效果影响更大，且答案正确性不影响蒸馏结果
将具备推理能力的大语言模型的思维轨迹蒸馏到小型非推理模型中已被证明有效。然而，关于蒸馏数据量与模型性能之间的缩放关系研究较少。本文研究了在两个小型非推理大语言模型上，蒸馏竞赛级代码能力的缩放趋势。我们验证了‘代码推理谷地’假说：下游代码竞赛性能随数据量增加先下降，随后以高于对数线性的速度持续上升。在识别该趋势后，我们在同一数据集上对模型在不同蒸馏阶段进行微调，以分析其学习阶段特征。结果显示，在低和中低数据量阶段，小模型从较简单代码题中获益远大于复杂题；同时发现，训练数据输出的正确性对蒸馏结果无显著影响。本工作推进了对代码推理蒸馏训练动态的理解，突破了直觉认知。
原文摘要 · Abstract (English)
Distilling the thinking traces of a Large Language Model (LLM) with reasoning capabilities into a smaller model has been proven effective. Yet, there is a scarcity of work done on how model performances scale with the quantity of distillation data. In this work, we study the scaling trend of distilling competitive coding skills on two small non-reasoning LLMs. We validate the hypothesis that there is a $\textit{valley of code reasoning}$: downstream performance on competitive coding first drops as data quantity increases, then it steadily increases in a sharper-than-log-linear fashion. Having identified the trend, we further fine-tune the models at two different distillation stages on the same data to ground conclusions on their respective learning phases. We learn that across stages in the low and medium-low data regimes, small models benefit significantly from easier coding questions than from harder ones. We also find that, surprisingly, the correctness of outputs in training data makes no difference to distillation outcomes. Our work represents a step forward in understanding the training dynamics of code reasoning distillation outside intuition
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。