用模型自生成代码微调，轻松提升编程能力
Embarrassingly Simple Self-Distillation Improves Code Generation
- 让模型自己生成代码样本，再用这些样本监督微调
- 在LiveCodeBench上从42.4%提升至55.3%通过率，难题效果更明显
- 方法简单通用，适配多种大模型和规模
大型语言模型能否仅凭自身输出的原始代码，在无验证器、无教师模型、无强化学习的情况下提升编程能力？我们以简单的自蒸馏（SSD）给出了肯定回答：在特定温度与截断配置下采样模型输出，再用这些样本进行标准监督微调。SSD将Qwen3-30B-Instruct在LiveCodeBench v6上的pass@1从42.4%提升至55.3%，增益集中在难题上；该方法在Qwen与Llama系列4B、8B、30B规模的指令型与思维型模型上均有效。我们进一步分析发现，性能提升源于解码过程中的精度-探索冲突，而SSD通过上下文相关方式重塑词元分布，在关键位置抑制干扰尾部以提升精度，同时保留探索所需多样性。这为提升大模型代码生成提供了一种互补的后训练方向。代码已开源。
原文摘要 · Abstract (English)
Can a large language model (LLM) improve at code generation using only its own raw outputs, without a verifier, a teacher model, or reinforcement learning? We answer in the affirmative with simple self-distillation (SSD): sample solutions from the model with certain temperature and truncation configurations, then fine-tune on those samples with standard supervised fine-tuning. SSD improves Qwen3-30B-Instruct from 42.4% to 55.3% pass@1 on LiveCodeBench v6, with gains concentrating on harder problems, and it generalizes across Qwen and Llama models at 4B, 8B, and 30B scale, including both instruct and thinking variants. To understand why such a simple method can work, we trace these gains to a precision-exploration conflict in LLM decoding and show that SSD reshapes token distributions in a context-dependent way, suppressing distractor tails where precision matters while preserving useful diversity where exploration matters. Taken together, SSD offers a complementary post-training direction for improving LLM code generation. Our code is available at https://github.com/apple/ml-ssd
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。