模仿人类学习过程,动态调整奖励以提升大模型推理能力。
Boosting LLM Reasoning via Human-Inspired Reward Shaping
- 分阶段设计奖励:错误时拓宽思路,正确时压缩表达。
- 在数学基准上,5个主流模型性能显著优于基线。
- 适合追求模型推理效率与可解释性的研究者使用。
强化学习结合可验证奖励(RLVR)已成为提升大语言模型推理能力的有前景范式。然而,现有奖励设计通常将探索与巩固视为单一过程,导致学习阶段动态纠缠,违背人类学习规律。人类学习中,面对未掌握问题时注重广泛探索以寻找可行解;而对于已掌握问题,则聚焦于推理浓缩与知识抽象,提炼简洁核心原则。受此启发,我们提出T2T(Thickening-to-Thinning)动态奖励框架,模拟人类学习过程。具体包括双阶段机制:(1)在错误尝试时,激励“加厚”以扩大搜索空间,探索新解法路径;(2)在正确后,转向“减薄”,施加长度惩罚以减少冗余,增强模型自信并凝练推理能力。在5个主流LLM上对MATH-500、AIME、AMC等数学基准的大量实验表明,T2T显著优于标准GRPO及近期基线,表现更优。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a promising paradigm for enhancing reasoning in Large Language Models (LLMs). However, existing reward formulations typically treat exploration and consolidation as a monolithic process, resulting in entangled stage-wise learning dynamics. This contradicts the natural learning behavior of human learners. In human learning, individuals adopt distinct behavioral patterns toward mastered versus unfamiliar problems. When confronting unmastered challenges, humans prioritize broad exploration to seek viable solutions. By contrast, for well-mastered problems, they focus instead on reasoning condensation and knowledge abstraction to distill concise underlying principles. Motivated by this gap, we introduce T2T(Thickening-to-Thinning), a dynamic reward framework inspired by human learning processes. Specifically, it implements a dual-phase mechanism: (1) On incorrect attempts, T2T incentivizes "thickening" to broaden the search space and explore novel solution paths; (2) Upon achieving correctness, it shifts to "thinning", imposing length penalties to discourage redundancy, thereby fostering model confidence and crystallizing reasoning capabilities. Extensive experiments on mathematical benchmarks (MATH-500, AIME, AMC) across 5 mainstream LLMs demonstrate that T2T significantly outperforms standard GRPO and recent baselines, achieving superior performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。