用教师模型输出增强标签，让小模型学得更准更丰富。
Logits-Based Finetuning
- 用教师模型的输出概率+真实答案构造训练目标
- 数学推理任务上最高提升22.7%，平均提高7.28%
- 适合需要高精度和语言多样性的轻量级模型训练
近年来，研发紧凑高效的大型语言模型成为研究热点。传统监督微调依赖单一真实标签，难以捕捉词元级依赖关系和语言多样性。为此，我们提出一种基于输出概率（logits）的微调框架，融合监督学习与知识蒸馏优势。通过结合教师模型输出与真实标签构建丰富训练目标，既保证正确性又保留语言多样性，提升训练可靠性与有效性。我们构建了包含120万条样本的大规模日志数据集，并训练了一系列聚焦科学领域的模型。实验表明，该方法在Mawps上准确率提升18%，在TabMWP上提升22.7%；在九个主流数学基准测试中，平均性能提升7.28%，持续优于先前的SFT模型。代码已开源：https://github.com/dvlab-research/Logits-Based-Finetuning。
原文摘要 · Abstract (English)
In recent years, developing compact and efficient large language models (LLMs) has emerged as a thriving area of research. Traditional Supervised Fine-Tuning (SFT), which relies on singular ground truth labels, often fails to capture token-level dependencies and linguistic diversity. To address these limitations, we propose a logits-based fine-tuning framework that integrates the strengths of supervised learning and knowledge distillation. Our approach constructs enriched training targets by combining teacher logits with ground truth labels, preserving both correctness and linguistic diversity. This ensures more reliable and effective training. We constructed a large-scale 1.2M logits dataset and trained a series of science-focused models. Experimental results demonstrate that our method achieves significant improvements, with accuracy gains of 18% on Mawps and 22.7% on TabMWP. Across nine widely used mathematical benchmarks, our method consistently outperforms prior SFT models, achieving an average improvement of 7.28%. Codes are available at https://github.com/dvlab-research/Logits-Based-Finetuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。