arXiv:2603.20799cs.CLcs.LG2026-03被引 1

RLVR训练不能自动提升大模型通用问答能力,需新方法改进思维质量。

RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution

  • 提出跨生成评估框架,量化模型推理过程质量
  • 发现RLVR在通用问答上思维效果显著低于可验证任务
  • 设计START方法分步训练思维与回答,提升综合表现

强化学习从可验证奖励(RLVR)能激发大语言模型的思考过程,显著提升其在可验证任务上的推理能力。普遍认为此类提升应迁移至通用问答(GQA),但这一假设尚未充分验证。为评估RLVR是否自动改善GQA性能,我们提出跨生成评估框架,通过将生成的推理内容输入不同能力的LLM来衡量中间推理质量。结果表明,模型在GQA任务上的思维有效性明显低于可验证任务,说明仍需额外针对GQA进行训练。进一步观察发现,直接对GQA进行强化学习的效果不如RLVR。我们推测,可验证任务要求强逻辑链以获得高奖励,而GQA常存在无需高质量思维即可获取高奖励的捷径。为此,我们提出简单方法:分离思维与回答训练(START),先仅用最终答案定义奖励训练思维过程。实验显示,START在多个GQA基准和强化学习算法下均提升了思维质量与最终答案准确率。

原文摘要 · Abstract (English)

Reinforcement learning from verifiable rewards (RLVR) stimulates the thinking processes of large language models (LLMs), substantially enhancing their reasoning abilities on verifiable tasks. It is often assumed that similar gains should transfer to general question answering (GQA), but this assumption has not been thoroughly validated. To assess whether RLVR automatically improves LLM performance on GQA, we propose a Cross-Generation evaluation framework that measures the quality of intermediate reasoning by feeding the generated thinking context into LLMs of varying capabilities. Our evaluation leads to a discouraging finding: the efficacy of the thinking process on GQA tasks is markedly lower than on verifiable tasks, suggesting that explicit training on GQA remains necessary in addition to training on verifiable tasks. We further observe that direct RL training on GQA is less effective than RLVR. Our hypothesis is that, whereas verifiable tasks demand robust logical chains to obtain high rewards, GQA tasks often admit shortcuts to high rewards without cultivating high-quality thinking. To avoid possible shortcuts, we introduce a simple method, Separated Thinking And Response Training (START), which first trains only the thinking process, using rewards defined on the final answer. We show that START improves both the quality of thinking and the final answer across several GQA benchmarks and RL algorithms.

大模型强化学习推理能力通用问答

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。