用单题批判性微调,仅5小时就让大模型推理能力大幅提升。
Unleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problem
- 基于单个问题生成多种解法,用教师模型提供详细批注进行微调。
- 7B模型仅用5小时训练,数学与逻辑题平均提升15%~16%,媲美20倍算力的强化学习。
- 方法简单通用,适合想高效提升推理能力的研究者和开发者。
我们发现像Qwen-Math、MiMo和Phi-4这样的强语言模型在预训练阶段就具备强大的推理潜力。虽然强化学习(RL)能显著提升其表现,但成本高且不稳定,一次性的RL也需数百GPU小时。这引出关键问题:能否更高效地释放这些大模型的推理能力?本文提出仅在一个问题上进行批判性微调(CFT),即可有效激发模型潜力。方法通过收集单个问题的多种模型解法,并由教师模型提供详细批注构建数据。我们在1.5B至14B参数的Qwen和Llama系列模型上进行微调,在多个推理任务中取得显著提升。例如,仅用5 GPU小时训练的Qwen-Math-7B-CFT,在六个数学基准上平均提升15%,三个逻辑推理基准提升16%,效果媲美或超过使用20倍算力的强化学习。消融实验表明,该方法对不同提示问题具有鲁棒性。结果证明,单次CFT是一种简单、通用且高效的释放现代大模型推理能力的方法。
原文摘要 · Abstract (English)
We have witnessed that strong LLMs like Qwen-Math, MiMo, and Phi-4 possess immense reasoning potential inherited from the pre-training stage. With reinforcement learning (RL), these models can improve dramatically on reasoning tasks. Recent studies have shown that even RL on a single problem can unleash these models' reasoning capabilities. However, RL is not only expensive but also unstable. Even one-shot RL requires hundreds of GPU hours. This raises a critical question: Is there a more efficient way to unleash the reasoning potential of these powerful base LLMs? In this work, we demonstrate that Critique Fine-Tuning (CFT) on only one problem can effectively unleash the reasoning potential of LLMs. Our method constructs critique data by collecting diverse model-generated solutions to a single problem and using teacher LLMs to provide detailed critiques. We fine-tune Qwen and Llama family models, ranging from 1.5B to 14B parameters, on the CFT data and observe significant performance gains across diverse reasoning tasks. For example, with just 5 GPU hours of training, Qwen-Math-7B-CFT show an average improvement of 15% on six math benchmarks and 16% on three logic reasoning benchmarks. These results are comparable to or even surpass the results from RL with 20x less compute. Ablation studies reveal the robustness of one-shot CFT across different prompt problems. These results highlight one-shot CFT as a simple, general, and compute-efficient approach to unleashing the reasoning capabilities of modern LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。