arXiv:2609.05198cs.AI2026-09

用8个难例训练大模型,效果媲美17000条数据。

What Matters in On-Policy Distillation? A Perspective on Data Efficiency and Data Selection

  • 只选难样本训练,避开简单样本干扰。
  • 长思维链提升模型推理能力,比高熵更关键。
  • 适合资源有限但需高效微调的场景。

在推理领域,基于策略的蒸馏(OPD)已成为增强大语言模型的常用后训练范式。然而,其数据驱动机制仍缺乏深入研究。本文通过实证分析数据效率与数据选择问题,发现仅用一个样本(1-shot OPD)训练时,所有样本均有效,且难样本表现更优。分析表明,学生模型的提升并非来自高词元熵,而是由难题自然生成的长思维链路径所驱动。长思维链有助于维持师生模型在长推理过程中的对齐,并学习短思维链中缺失的关键思考模式,如反思(如“或者”)。基于此,我们提出一种简单数据筛选方法:仅使用难例训练,包括超出教师能力的“不可解”样本。在4个1.5B至7B模型上的实验显示,仅用8个精选难例训练,性能即可达到17K数据集基线水平。

原文摘要 · Abstract (English)

On-Policy Distillation (OPD) has emerged as a widely adopted post-training paradigm for enhancing large language models in reasoning domains. However, the data-centric mechanisms in OPD remain relatively underexplored. This paper presents a empirical study of data efficiency and data selection in OPD. We begin by investigating an extreme setting: training OPD on only one example, namely 1-shot OPD. Surprisingly, we find that 1-shot OPD is consistently effective across all sampled training examples and harder examples often yield superior performance gain. We next investigate what actually drives the student model's improvement in the training data. Our analysis reveals that the improvement is not driven by high token entropy, but the longer CoT paths which hard problems naturally generate. Training on longer CoT can help maintain closer alignment with the teacher over a long reasoning horizon, and learn critical thinking patterns usually missing in short CoTs, such as reflection (e.g., ``Alternatively''). Based on these insights, we propose a simple data selection method that selects only hard examples for training, where even ``unsolvable'' examples that completely exceed the teacher's capability can be successfully used. Our experiments conducted on four models ranging from 1.5B to 7B show that training the student model on only 8 selected hard examples matches the performance of the 17K dataset baseline.

模型蒸馏数据效率思维链难例筛选

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。