用强化学习提升大模型推理质量,让思考过程更有效。
Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
- 设计新奖励机制,评估推理链对答案的真正帮助。
- 70亿参数模型400步训练即达GPT-o3-mini水平,置信度提升30%。
- 适用于数学、逻辑推理任务,适合想提升模型思维能力的研究者。
强化学习(RL)已成为增强大语言模型(LLM)推理能力的主要方法。然而,当前数学与编程基准中常用的基于规则的奖励函数仅评估答案格式与正确性,无法判断推理链是否真正提升了答案质量。此外,任务特定训练难以控制逻辑深度,可能无法揭示模型的真实推理能力。本文提出动态推理效率奖励(DRER),一种可插拔的强化学习奖励框架,重构奖励与优势信号:(i) 推理质量奖励为显著提高正确答案概率的推理链赋予细粒度奖励,直接激励有益的推理过程;(ii) 动态长度优势通过衰减偏离验证集阈值的响应优势,稳定训练过程。为支持严格评估,我们发布Logictree——一个动态构建的演绎推理数据集,兼具训练与评测功能。实验表明,使用DRER的70亿参数模型在Logictree上仅需400次训练即达到GPT-o3-mini水平,且推理增强答案的平均置信度提升30%。模型还在多种逻辑推理数据集及数学竞赛基准AIME24上展现良好泛化能力。结果揭示了强化学习如何塑造推理行为,并为提升大模型形式化推理能力提供可行路径。所有代码与数据已开源于https://github.com/Henryhe09/DRER。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has recently become the dominant paradigm for strengthening the reasoning abilities of large language models (LLMs). Yet the rule-based reward functions commonly used on mathematical or programming benchmarks assess only answer format and correctness, providing no signal as to whether the induced Chain-of-Thought (CoT) actually improves the answer. Furthermore, such task-specific training offers limited control over logical depth and therefore may fail to reveal a model's genuine reasoning capacity. We propose Dynamic Reasoning Efficiency Reward (DRER) -- a plug-and-play RL reward framework that reshapes both reward and advantage signals. (i) A Reasoning Quality Reward assigns fine-grained credit to those reasoning chains that demonstrably raise the likelihood of the correct answer, directly incentivising the trajectories with beneficial CoT tokens. (ii) A Dynamic Length Advantage decays the advantage of responses whose length deviates from a validation-derived threshold, stabilising training. To facilitate rigorous assessment, we also release Logictree, a dynamically constructed deductive reasoning dataset that functions both as RL training data and as a comprehensive benchmark. Experiments confirm the effectiveness of DRER: our 7B model attains GPT-o3-mini level performance on Logictree with 400 trianing steps, while the average confidence of CoT-augmented answers rises by 30%. The model further exhibits generalisation across diverse logical-reasoning datasets, and the mathematical benchmark AIME24. These results illuminate how RL shapes CoT behaviour and chart a practical path toward enhancing formal-reasoning skills in large language models. All code and data are available in repository https://github.com/Henryhe09/DRER.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。