先判断后生成,让大模型更高效地解题
JudgeRLVR: Judge First, Generate Second for Efficient Reasoning
- 先训练模型判断答案对错,再用其指导生成
- 数学题平均准确率提升3.7点,长度减少42%
- 适合追求高效推理的AI系统开发者
强化学习结合可验证奖励(RLVR)已成为大语言模型推理的标准范式。然而,仅优化最终答案正确性常导致模型陷入冗长无序的试错探索,而非结构化规划。虽然长度惩罚等启发式约束可减少冗余,但常误删关键推理步骤,造成效率与验证之间的权衡难题。本文提出JudgeRLVR:两阶段判别-生成框架。第一阶段训练模型判断具有可验证答案的解法是否正确;第二阶段以该判别模型初始化的生成式RLVR进行微调。相比使用相同数学领域数据的原始RLVR,在Qwen3-30B-A3B上,该方法在域内数学任务中实现约+3.7分准确率提升,平均生成长度降低42%;在域外基准上,平均准确率提升约4.5分,展现更强泛化能力。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for reasoning in Large Language Models. However, optimizing solely for final-answer correctness often drives models into aimless, verbose exploration, where they rely on exhaustive trial-and-error tactics rather than structured planning to reach solutions. While heuristic constraints like length penalties can reduce verbosity, they often truncate essential reasoning steps, creating a difficult trade-off between efficiency and verification. In this paper, we argue that discriminative capability is a prerequisite for efficient generation: by learning to distinguish valid solutions, a model can internalize a guidance signal that prunes the search space. We propose JudgeRLVR, a two-stage judge-then-generate paradigm. In the first stage, we train the model to judge solution responses with verifiable answers. In the second stage, we fine-tune the same model with vanilla generating RLVR initialized from the judge. Compared to Vanilla RLVR using the same math-domain training data, JudgeRLVR achieves a better quality--efficiency trade-off for Qwen3-30B-A3B: on in-domain math, it delivers about +3.7 points average accuracy gain with -42\% average generation length; on out-of-domain benchmarks, it delivers about +4.5 points average accuracy improvement, demonstrating enhanced generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。