通过几何重构拓展大模型推理空间，突破原有能力边界。
Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization
- 提出曼达托重塑策略优化（MRPO），分两阶段重构模型推理空间。
- 40亿参数模型在数学任务上超越320亿参数的Qwen3，性能领先。
- 适合研究大模型推理能力本质与强化学习新范式的研究者。
基于可验证奖励的强化学习（RLVR）在提升大语言模型（LLM）推理能力方面表现卓越。然而，近期研究质疑强化学习是否真正扩展了推理能力，还是仅对预训练模型固有的低秩偏置流形进行对齐，认为探索仍受限于该流形。本文挑战这一可访问性边界假设，证明可通过定向几何干预从根本上拓展潜在推理空间。我们提出曼达托重塑策略优化（MRPO），一个旨在重构LLM推理空间的几何框架。该方法分为两阶段：首先使用谱正交探索（SOE）将策略初始化推入偏置流形的零空间；其次在策略优化目标中引入有效秩正则化项，以激励发现并维持高维推理轨迹，对抗标准强化学习的熵减倾向。实证表明，40亿参数的MRPO方法在数学任务上达到当前最优性能，显著优于更大规模的模型（如Qwen3-32B），并将能力边界拓展至标准GRPO之外。代码已公开于https://anonymous.4open.science/r/MRPO-D57B/
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs). However, recent studies question whether RL genuinely expands reasoning capacity or merely aligns existing latent capabilities, arguing that exploration remains confined within the pre-trained model's low-rank bias manifold. In this work, we challenge this accessibility boundary hypothesis by demonstrating that the latent reasoning space can be fundamentally expanded through targeted geometric interventions. We propose Manifold-Reshaping Policy Optimization (MRPO), a geometric framework designed to fundamentally restructure the inference space of LLMs. MRPO operates in two stages: first, we employ Spectral Orthogonal Exploration (SOE) to eject the policy initialization into the null space of the bias manifold; second, we integrate an Effective Rank regularization term into the policy optimization objective. This approach incentivizes the discovery and maintenance of high-dimensional reasoning trajectories against the entropy-reducing tendency of standard RL. Empirically, our 4B-parameter method achieves state-of-the-art performance on mathematical tasks, significantly outperforming larger models (e.g., Qwen3-32B) and expanding the capability boundary beyond standard GRPO. Our code is available at https://anonymous.4open.science/r/MRPO-D57B/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。