聚焦推理开头,让大模型更高效地思考。
Well Begun, Half Done: Reinforcement Learning with Prefix Optimization for LLM Reasoning
- 只优化推理开头部分的关键词,提升学习效率
- 仅用26%的训练数据,准确率提升18%
- 适合想提升大模型逻辑推理能力的研究者
强化学习结合可验证奖励(RLVR)显著提升了大语言模型(LLMs)的推理能力。现有方法通常对所有生成词元进行统一训练,却忽略了哪些词元(如前缀)真正影响推理结果。这种平均优化策略浪费大量资源在低回报词元上,抑制了高回报词元的改进空间,降低整体训练效率。为此,本文提出一种新型RLVR方法——渐进式前缀策略优化(PPPO),强调生成输出中前缀部分的重要性。受人类思维中路径依赖理论启发,我们发现大模型推理存在‘初始锁定效应’(BLE),即早期思考极大限制后续推理轨迹。PPPO通过聚焦前缀推理过程,实现对后续推理的正向引导。为提升高质量起始推理的学习效果,引入两种策略:(a) 渐进式前缀保留,逐步提高训练中保留的前缀比例;(b) 续写累积奖励,通过采样多个续写序列并累加得分作为奖励信号,缓解奖励偏差。在多种推理任务上的实验表明,相比代表性RLVR方法,PPPO在仅使用26.17%训练词元的情况下,准确率提升达18.02%。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capability of Large Language Models (LLMs). Current RLVR approaches typically conduct training across all generated tokens, but neglect to explore which tokens (e.g., prefix tokens) actually contribute to reasoning. This uniform training strategy spends substantial effort on optimizing low-return tokens, which in turn impedes the potential improvement from high-return tokens and reduces overall training effectiveness. To address this issue, we propose a novel RLVR approach called Progressive Prefix-token Policy Optimization (PPPO), which highlights the significance of the prefix segment of generated outputs. Specifically, inspired by the well-established human thinking theory of Path Dependence, where early-stage thoughts substantially constrain subsequent thinking trajectory, we identify an analogous phenomenon in LLM reasoning termed Beginning Lock-in Effect (BLE). PPPO leverages this finding by focusing its optimization objective on the prefix reasoning process of LLMs. This targeted optimization strategy can positively influence subsequent reasoning processes, and ultimately improve final results. To improve the learning effectiveness of LLMs on how to start reasoning with high quality, PPPO introduces two training strategies: (a) Progressive Prefix Retention, which shapes a progressive learning process by increasing the proportion of retained prefix tokens during training; (b) Continuation Accumulated Reward, which mitigates reward bias by sampling multiple continuations for one prefix token sequence, and accumulating their scores as the reward signal. Extensive experimental results on various reasoning tasks demonstrate that our proposed PPPO outperforms representative RLVR methods, with the accuracy improvements of 18.02% on only 26.17% training tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。