用强化学习让大模型学会深度思考，小规划器即可显著提升推理能力。
rSIM: Incentivizing Reasoning Capabilities of LLMs via Reinforced Strategy Injection
- 通过多智能体强化学习，用小规划器动态注入推理策略。
- 0.5B模型经rSIM后超越14B原模型的推理表现。
- 规划器一次训练可通用，支持持续学习和跨任务迁移。
大型语言模型（LLMs）通过强化学习（RL）可进化为推理语言模型（RLMs），其核心特征是在思维链（CoTs）中产生“顿悟”时刻，开始运用自我反思、深度思考等策略。本文提出一种新型强化策略注入机制（rSIM），通过一个小规划器（leader agent）与大模型（follower agent）在领导-跟随框架下联合训练，利用规则奖励实现自适应策略注入，使任意LLM具备成为RLM的能力。实验表明，rSIM使Qwen2.5-0.5B模型实现推理能力跃升，显著优于未优化的Qwen2.5-14B。该规划器具有强泛化性：仅需一次训练即可作为插件，广泛提升现有模型的推理表现，并支持跨任务持续学习，逐步增强规划能力，适用于多样化问题场景。
原文摘要 · Abstract (English)
Large language models (LLMs) are post-trained through reinforcement learning (RL) to evolve into Reasoning Language Models (RLMs), where the hallmark of this advanced reasoning is ``aha'' moments when they start to perform strategies, such as self-reflection and deep thinking, within chain of thoughts (CoTs). Motivated by this, this paper proposes a novel reinforced strategy injection mechanism (rSIM), that enables any LLM to become an RLM by employing a small planner to guide the LLM's CoT through the adaptive injection of reasoning strategies. To achieve this, the planner (leader agent) is jointly trained with an LLM (follower agent) using multi-agent RL (MARL), based on a leader-follower framework and straightforward rule-based rewards. Experimental results show that rSIM enables Qwen2.5-0.5B to become an RLM and significantly outperform Qwen2.5-14B. Moreover, the planner is generalizable: it only needs to be trained once and can be applied as a plug-in to substantially improve the reasoning capabilities of existing LLMs. In addition, the planner supports continual learning across various tasks, allowing its planning abilities to gradually improve and generalize to a wider range of problems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。