让模型在强化学习前通过自蒸馏学会推理技能，提升后续泛化与鲁棒性。
SkillFactory: Self-Distillation For Learning Cognitive Behaviors
- 用模型自身生成的推理轨迹重构训练数据，预习认知技能
- 强化学习后模型在更难任务上表现更好，即使初始性能较低
- 适合想提升模型推理鲁棒性的研究人员
推理模型依赖长链思维中的多种认知技能，如答案验证、回溯、换法重试等。已有研究证明，若基础语言模型具备这些技能，通过强化学习（RL）可进一步学会利用它们。但如何让不具备这些技能的基础模型掌握？本文提出SkillFactory方法，在强化学习前的监督微调（SFT）阶段，通过重组模型自身生成的推理轨迹，构建“银级”训练样本，实现对认知技能的初步学习。该方法不依赖更强模型的蒸馏，而是利用模型自我生成的、虽不完美但有效的推理路径。评估表明：（1）从SkillFactory SFT初始化开始，模型在强化学习后能更好地泛化到更难的任务变体，尽管初始性能较差；（2）模型确实在推理中使用了这些认知技能；（3）经强化学习后的SkillFactory模型在跨域任务上更具鲁棒性，优于仅基于基础模型的强化学习模型。结果表明，强化学习前引入归纳偏置有助于模型习得稳健的认知技能。
原文摘要 · Abstract (English)
Reasoning models leveraging long chains of thought employ various cognitive skills, such as verification of their answers, backtracking, retrying by an alternate method, and more. Previous work has shown that when a base language model exhibits these skills, training that model further with reinforcement learning (RL) can learn to leverage them. How can we get models to leverage skills that aren't exhibited by base models? Our work, SkillFactory, is a method for fine-tuning models to roughly learn these skills during a supervised fine-tuning (SFT) stage prior to RL. Our approach does not rely on distillation from a stronger model, but instead uses samples from the model itself, rearranged to provide training data in the format of those skills. These "silver" SFT traces may be imperfect, but are nevertheless effective for priming a model to acquire skills during RL. Our evaluation shows that (1) starting from SkillFactory SFT initialization helps a model to generalize to harder variants of a task post-RL, despite lower performance pre-RL;(2) cognitive skills are indeed used by the model; (3) RLed SkillFactory models are more robust to regression on out-of-domain tasks than RLed base models. Our work suggests that inductive biases learned prior to RL help models learn robust cognitive skill use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。