通过智能诊断与修复低效推理树,提升大模型数学推理能力。
CATPO: Critique-Augmented Tree Policy Optimization
- 用树结构信息度量评分筛选有效推理路径。
- 对全失败树生成语言批评并修复路径,恢复训练信号。
- 仅更新高信息量树的梯度,显著提升训练效率与效果。
基于可验证奖励的强化学习(RLVR)已成为提升大语言模型(LLM)推理能力的主要范式。近期基于树的方法如TreeRPO通过树状轨迹采样,在无需额外奖励模型的情况下获得密集的逐步奖励信号。然而,并非所有树都具有同等信息量:所有叶子成功、全部失败,或策略已准确预测奖励分布的树,对梯度更新贡献极小,造成计算浪费。本文提出CATPO(Critique-Augmented Tree Policy Optimization),在树层面诊断并解决这一问题。CATPO首先通过树信息度量分数F(T)对每棵树打分,该分数结合叶子结果多样性与策略-奖励去相关性,且不增加额外计算开销。对于全失败的树,CATPO采用批判引导修复:定位最浅层失败点,生成自然语言批评,并嫁接优化后的延续路径以恢复训练信号。最后,使用信息度量加权损失,按归一化得分缩放每棵树的梯度贡献,使参数更新集中在最具信息量的树上,同时保持整体梯度幅度。在使用MATH数据集训练的Qwen2.5-Math-1.5B模型上,CATPO在四个基准测试(AIME24、MATH-500、OlympiadBench、MinervaMath)中达到37.5%的宏平均准确率,相比TreeRPO提升1.9%,相比GRPO提升4.8%。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has become a dominant paradigm for improving the reasoning capabilities of large language models (LLMs). Recent tree-based methods such as TreeRPO extend flat trajectory sampling with tree-structured rollouts to obtain dense, step-level reward signals without a separate process reward model. However, not all trees are equally informative: trees where all leaves succeed, all leaves fail, or the policy already predicts the reward distribution contribute little to gradient updates, wasting compute. We introduce CATPO (Critique-Augmented Tree Policy Optimization), which diagnoses and addresses this waste at the tree level. CATPO first scores each tree via a tree informativeness score, F(T), combining leaf-outcome diversity with policy-reward decorrelation at zero extra compute. For dead-wrong trees where all branches fail, CATPO applies critique-guided healing: it locates the shallowest failure point, generates a natural-language critique, and grafts refined continuations to recover training signal. Finally, an informativeness-weighted loss scales each tree's gradient contribution by its normalized score, concentrating parameter updates on the most informative trees while preserving overall gradient magnitude. Experiments on Qwen2.5-Math-1.5B trained with the MATH dataset show that CATPO achieves 37.5% macro accuracy across four benchmarks (AIME24, MATH-500, OlympiadBench, and MinervaMath), improving over TreeRPO by 1.9% and GRPO by 4.8%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。