仅用一个训练样例，大模型也能实现接近全数据的强化学习优化效果。
Rethinking On-Policy Distillation of Large Language Models II: One Training Example
- 用单个查询进行在线策略蒸馏，仍能持续优化数百步。
- 单个查询覆盖71.5%的关键状态，16个不同查询达98.9%覆盖率。
- 当前方法更缺算法效率而非数据量，适合关注训练效率的研究者。
在线策略蒸馏（OPD）结合学生模型生成的轨迹与教师模型的密集令牌级监督。现有研究多聚焦算法行为，忽视训练数据的作用。本文在数据极简条件下研究该问题，仅用单个查询进行训练。结果表明，单样本OPD可持续优化数百步，且在多个任务领域和模型家族中恢复了大部分全数据训练的收益。通过分析训练中访问的状态分布及学生与教师对齐速率，发现单个查询已覆盖71.5%的全数据关键状态，其中大部分在前100步内完成。增加语义多样性的查询可同步提升状态覆盖率与验证准确率，16个查询达到98.9%，逼近全数据性能。然而，无论使用单个查询还是全数据集，学生对齐速度均相似下降，且即使固定状态集也需数百步才能吸收。因此，OPD本质是数据过饱但算法饥饿。其轨迹快速暴露广泛监督信号，但学生吸收过程却渐缓。该状态覆盖结论扩展至多教师OPD，每领域16个语义多样查询即可匹配全数据性能。进一步测试显示，内容稀疏模板和跨域野聊查询亦趋近真实查询基线。这表明任务内容与诱导状态覆盖可分离。研究呼吁未来工作应聚焦OPD的步骤效率，并重新审视其近期前沿后训练成功背后的机制与数据角色。
原文摘要 · Abstract (English)
On-policy distillation (OPD) combines student-generated rollouts with dense token-level supervision from a teacher. Existing work has mainly studied its algorithmic behavior, leaving the role of training data unclear. We examine this role at the data-minimal limit by training on a single query. One-shot OPD keeps improving for hundreds of steps and recovers most of full-data OPD's gain across task domains and model families. We explain this result through the states visited during training and the rate at which the student aligns with the teacher. We measure \emph{state coverage}, the fraction of the states full-data OPD visits that a query set's rollouts reach. A single query already reaches \(71.5\%\), most of it within the first 100 steps. Adding semantically distinct queries raises coverage and validation accuracy together, until 16 queries reach \(98.9\%\) and match full-data training. Yet alignment slows at a similar pace whether OPD trains on one query or the whole dataset, and even a fixed set of states takes hundreds of steps to absorb. OPD is therefore data-overfed but algorithm-starved. Its rollouts quickly expose broad supervision, while the student absorbs that supervision increasingly slowly. The state-coverage result extends to multi-teacher OPD, where 16 semantically diverse queries per domain match full-data MOPD. As a further stress test, content-light templates and off-domain WildChat queries also approach the real-query baseline. Task content and induced state coverage can therefore come apart. We hope these findings direct future work toward the step efficiency of OPD, and prompt a re-examination of the data and the mechanisms behind its recent successes in frontier post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。