用强化学习训练大模型分而治之推理,提升复杂问题求解能力
Training LLMs for Divide-and-Conquer Reasoning Elevates Test-Time Scalability
- 通过强化学习让模型自主拆解问题并分步求解
- 在竞赛级基准上性能超越传统链式思维8.6%(Pass@1)
- 适合需要高扩展性推理的复杂任务场景
大语言模型(LLMs)通过逐步的思维链(CoT)推理展现出强大能力。然而,在模型能力极限下,CoT往往不足,且其严格顺序性限制了测试时的可扩展性。分而治之(DAC)推理将复杂问题分解为子问题,有助于更高效地探索解决方案。尽管前景广阔,我们的分析揭示通用后训练与DAC推理之间存在根本性不匹配,限制了模型充分发挥潜力。为此,我们提出一种端到端强化学习(RL)框架,增强模型的DAC推理能力:每一步中,策略将问题分解为子问题,顺序求解,并基于子问题结果重构原问题解答,分解与求解均纳入强化学习训练。在同等训练条件下,该框架使模型达到更高性能上限和更强测试时可扩展性,在竞赛级基准上,相比CoT提升8.6%(Pass@1)和6.3%(Pass@32)。
原文摘要 · Abstract (English)
Large language models (LLMs) have demonstrated strong reasoning capabilities through step-by-step chain-of-thought (CoT) reasoning. Nevertheless, at the limits of model capability, CoT often proves insufficient, and its strictly sequential nature constrains test-time scalability. A potential alternative is divide-and-conquer (DAC) reasoning, which decomposes a complex problem into subproblems to facilitate more effective exploration of the solution. Although promising, our analysis reveals a fundamental misalignment between general-purpose post-training and DAC-style inference, which limits the model's capacity to fully leverage this potential. To bridge this gap and fully unlock LLMs' reasoning capabilities on the most challenging tasks, we propose an end-to-end reinforcement learning (RL) framework to enhance their DAC-style reasoning capacity. At each step, the policy decomposes a problem into a group of subproblems, solves them sequentially, and addresses the original one conditioned on the subproblem solutions, with both decomposition and solution integrated into RL training. Under comparable training, our DAC-style framework endows the model with a higher performance ceiling and stronger test-time scalability, surpassing CoT by 8.6% in Pass@1 and 6.3% in Pass@32 on competition-level benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。