给大模型讲清任务规则,能显著提升强化学习微调效果。
A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models
- 在提示中加入奖励规则作为内在动机,引导模型理解优化目标。
- 相比基线,性能显著提升,且在动机与奖励一致时效果更佳。
- 适合需要高效强化学习微调的复杂推理任务研究者。
基于可验证奖励的强化学习(RLVR)已成为大型推理模型解决复杂任务的强大范式。然而,当前方法效率仍不足,因依赖试错机制,模型需大量生成响应并从碎片化奖励信号中学习,难以察觉整体奖励模式。得益于可验证奖励支持自然语言描述奖励函数,且大模型具备强上下文学习能力,我们探索是否可在强化微调过程中让模型获得任务的「动机」——即对奖励函数的认知,如同人类学习时了解规则。本文提出动机增强型强化微调(MeRF),通过在提示中直接注入奖励说明,作为模型的上下文动机,使其明确优化目标。该方法利用大模型的上下文学习能力,使生成行为与优化目标对齐,同时激发内生动机与外部奖励的协同作用。实验表明,MeRF显著优于传统RLVR基线;消融实验证明,上下文动机与外部奖励越一致,效果越好;模型亦可通过强化微调适应误导性动机。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards~(RLVR) has emerged as a powerful learn-to-reason paradigm for large reasoning models to tackle complex tasks. However, the current RLVR paradigm is still not efficient enough, as it works in a trial-and-error manner. To perform better, the model needs to explore the reward space by numerously generating responses and learn from fragmented reward signals, blind to the overall reward patterns. Fortunately, verifiable rewards make the natural language description of the reward function possible, and meanwhile, LLMs have demonstrated strong in-context learning ability. This motivates us to explore if large reasoning models can benefit from a \textbf{motivation} of the task, \textit{i.e.}, awareness of the reward function, during the reinforcement finetuning process, as we humans sometimes do when learning. In this paper, we introduce \textit{\textbf{M}otivation-\textbf{e}nhanced \textbf{R}einforcement \textbf{F}inetuning}~(\textbf{MeRF}), an intuitive yet effective method enhancing reinforcement finetuning of LLMs by involving \emph{``telling LLMs rules of the game''}. Specifically, \textbf{MeRF} directly injects the reward specification into the prompt, which serves as an in-context motivation for the model to be aware of the optimization objective. This simple modification leverages the in-context learning ability of LLMs, aligning generation with optimization, thereby incentivizing the model to generate desired outputs from both inner motivation and external reward. Empirical evaluations demonstrate that \textbf{MeRF} achieves substantial performance gains over the RLVR baseline. Moreover, ablation studies show that MeRF performs better with greater consistency between the in-context motivation and the external reward function, while the model also demonstrates an ability to adapt to misleading motivations through reinforcement finetuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。