RLVR虽提升数学题得分，却常靠捷径而非真正推理。
Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
- 用可验证奖励训练模型解决组合问题
- 多组实验显示得分提升但策略仍肤浅
- 适合关注模型是否真会解题的研究者
数学推理是大语言模型的核心挑战，不仅需要正确答案，还需可靠的推理过程。基于可验证奖励的强化学习（RLVR）被视为提升该能力的有前景方法，但其能否真正促进深层推理尚不明确。我们针对两个具有完全可验证解的组合问题——活动调度和最长递增子序列——使用精心构建的数据集（含唯一最优解），在多种奖励设计下进行实验。结果显示，尽管RLVR提升了评估指标，但多数情况下是通过强化表面启发式策略，而非掌握新的推理方法。这揭示了RLVR泛化能力的局限性，强调需设计能区分真实推理与捷径利用的基准，并提供可信的进展度量。代码已开源：https://github.com/xashru/rlvr-seq-generalization。
原文摘要 · Abstract (English)
Mathematical reasoning is a central challenge for large language models (LLMs), requiring not only correct answers but also faithful reasoning processes. Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising approach for enhancing such capabilities; however, its ability to foster genuine reasoning remains unclear. We investigate RLVR on two combinatorial problems with fully verifiable solutions: \emph{Activity Scheduling} and the \emph{Longest Increasing Subsequence}, using carefully curated datasets with unique optima. Across multiple reward designs, we find that RLVR improves evaluation metrics but often by reinforcing superficial heuristics rather than acquiring new reasoning strategies. These findings highlight the limits of RLVR generalization, emphasizing the importance of benchmarks that disentangle genuine mathematical reasoning from shortcut exploitation and provide faithful measures of progress. Code available at https://github.com/xashru/rlvr-seq-generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。