通过训练时分解难题,让模型在无奖励情况下也能学会解决复杂问题。
TD-Grokking: Learning from Zero-Reward Problems by Training-Time Decomposition

- 将难题递归拆解为可验证的子问题,形成带奖励的层级树
- 在数学和医疗任务上显著优于基线方法,提升模型性能
- 适合解决无反馈信号的复杂推理任务,如零奖励难题
大型语言模型在推理任务中取得显著进展,主要得益于后训练范式,尤其是可验证奖励的强化学习(RLVR)。然而,一个关键瓶颈仍存在:当所有采样的推理路径均导致失败结果时,即零奖励问题,RLVR无法提供优化信号。现有方法如密集过程监督、部分奖励分配或前缀引导探索,受限于任务本身或未充分赋予策略模型解决原始不可解问题的能力。为此,我们提出TD-Grokking,一种针对零奖励问题的训练时分解框架。该方法将难以处理的根问题递归分解为自包含且可验证的子问题,形成层级树结构,其中可解的叶节点提供非零奖励。在数学与医疗任务上的评估显示,TD-Grokking优于原始GRPO及所有基线方法。结合详细分析,结果证实训练时分解能有效将零奖励样本转化为可用训练信号,实现稳定性能提升。代码与数据集见https://anonymous.4open.science/r/TD-Grokking-6567/。
原文摘要 · Abstract (English)
Large language models (LLMs) have made remarkable progress in reasoning tasks, largely driven by post-training paradigms, especially reinforcement learning with verifiable rewards (RLVR). However, a critical bottleneck persists: RLVR fails on highly challenging zero-reward problems, where all sampled reasoning trajectories yield uniformly failed outcomes, providing no optimization signal to drive model improvement. Prior efforts to address this limitation, such as dense process supervision, partial reward assignment, or prefix-guided exploration, suffer from inherent task constraints or do not fully equip the policy model with the capabilities necessary to solve the original intractable problems. To address this, we propose TD-Grokking, a training-time decomposition framework for zero-reward problems. It recursively decomposes intractable root problems into self-contained, verifiable subproblems, forming hierarchical trees where solvable leaves provide non-zero rewards. Evaluations on mathematical and medical tasks show that TD-Grokking outperforms vanilla GRPO as well as all baseline approaches. Together with detailed analysis, these results confirm that training-time decomposition effectively converts zero-reward examples into usable training signals, enabling consistent performance gains. Our code and datasets are available at https://anonymous.4open.science/r/TD-Grokking-6567/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。