通过自适应分解能力，让大模型在强化学习中更高效地推理复杂问题。
Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning
- 用强化学习训练分解器，自动拆解复杂问题为简单子问题。
- 在无教师模型情况下，使推理模型准确率提升12.7%以上。
- 可插件式适配不同强化学习算法，适合需要高效推理的场景。
基于可验证奖励的强化学习（RLVR）在提升大语言模型（LLM）推理能力方面展现出巨大潜力。然而，由于RLVR过程中信息有限，模型只能进行大量盲目探索，常在复杂任务上失败。为此，我们提出A$^2$D方法——自适应能力分解，以增强RLVR的有效性。首先，我们通过无蒸馏的RLVR训练一个分解器，使其能将复杂问题拆解为一系列简单子问题；随后，利用该分解器对训练数据中的每个问题标注子问题，并在子问题引导下使用RLVR训练推理模型。我们对比了多种基线方法，验证了A$^2$D的有效性。进一步发现，该方法可作为即插即用模块，适用于不同RLVR算法。此外，我们分析了分解器的行为，揭示了RLVR如何影响其性能，以及何种引导方式更有利于提升推理模型的探索与利用能力。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has shown great potential to enhance the reasoning ability of large language models (LLMs). However, due to the limited amount of information provided during the RLVR process, the model can only engage in largely blind exploration, which often results in failure on challenging problems. To provide additional information for the RLVR process without relying on a teacher model, we propose A$^2$D, an Adaptive Ability Decomposing method for enhancing the effectiveness of RLVR. Specifically, we first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions. Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance. To better understand A$^2$D, we first compare its performance with competitive baselines, showing its effectiveness. Next, we observe that our method functions as a plug-and-play module that can be applied to different RLVR algorithms. Furthermore, we conduct an analysis of the decomposer, revealing how the RLVR process affects its performance and behavior, and which type of guidance is better suited for enhancing the reasoner's exploration and exploitation abilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。