RLVR让模型学会高效回溯，推理效率远超传统监督微调。
Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently
- 将思维链建模为图上的寻路问题，对比强化学习与监督微调。
- SFT无法学会有效回溯，而RLVR能仅凭结果奖励实现高效回溯。
- 适用于需要高效推理的复杂任务，如数学证明与逻辑推演。
近期大语言模型进展表明，对预训练基础模型进行强化微调可显著提升推理性能。本文从理论上分析为何强化微调相比纯监督微调（SFT）能带来更好的推理能力。我们将思维链（CoT）推理建模为图上的路径搜索问题，比较强化学习与可验证奖励（RLVR）方法与传统SFT。理论证明：在仅用最优最短路径训练且无负例的情况下，SFT无法学习如何高效回溯；而经由RLVR训练的模型仅凭最终结果奖励即可学会从死胡同中高效回溯。这导致两者在推理阶段计算量上存在指数级差异，证明了RLVR能使模型学会识别推理链中的困难决策点，从而更优地分配推理阶段的计算资源。最后，我们发现RLVR模型的推理轨迹可被提炼，用于训练基础模型实现高效回溯。
原文摘要 · Abstract (English)
Recent advances in large language models (LLMs) have demonstrated that reinforcement fine-tuning of pretrained base models can lead to significant gains in reasoning performance at inference time. In this work, we theoretically analyze why reinforcement fine-tuning induces better reasoning ability than purely supervised fine-tuning (SFT) methods. We model chain-of-thought (CoT) reasoning as a pathfinding problem on graphs and compare the popular method of reinforcement learning with verifiable rewards (RLVR) against traditional SFT. We prove that SFT, when trained on golden shortest paths without negative examples, fails to learn how to efficiently backtrack. In contrast, an RLVR-trained model can learn how to efficiently backtrack from dead ends using only outcome reward. This leads to an exponential separation in inference-time compute between the two methods, and demonstrates that RLVR leads the model to learn the location of difficult decisions in a reasoning chain, ultimately allowing for better allocation of inference-time compute. Finally, we show that the reasoning traces of an RLVR model can be distilled to train a base model to backtrack efficiently as well.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。