让大模型学会可迁移的推理,不只看答案对错
Towards Generalizable Reasoning: Group Causal Counterfactual Policy Optimization for LLM Reasoning
- 基于反事实因果视角,设计过程有效性奖励机制
- 在多个基准上显著提升推理泛化能力
- 适合需要可靠推理过程的AI系统开发者
大语言模型在复杂任务中表现出色,但现有奖励机制仅关注最终答案正确性,忽视推理过程:逻辑合理却答错的路径得分低,而侥幸猜对的错误逻辑可能被高分奖励,影响推理泛化。从因果视角看,同一问题的多候选推理可视为一组反事实实验。为此,我们提出群体因果反事实策略优化(Group Causal Counterfactual Policy Optimization),显式训练模型学习可迁移的推理模式。该方法设计一种周期性因果反事实奖励,同时捕捉:(i) 稳定性——推理步骤引发的答案分布在反事实扰动下保持稳定;(ii) 有效性——确保推理策略具有足够变异性以跨问题迁移。由此构建词级别优势,并优化策略,促使模型偏好过程有效且反事实鲁棒的推理模式。在多个基准上的广泛实验验证了其优越性。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at complex tasks with advances in reasoning capabilities. However, existing reward mechanisms remain tightly coupled to final correctness and pay little attention to the underlying reasoning process: trajectories with sound reasoning but wrong answers receive low credit, while lucky guesses with flawed logic may be highly rewarded, affecting reasoning generalization. From a causal perspective, we interpret multi-candidate reasoning for a fixed question as a family of counterfactual experiments with theoretical supports. Building on this, we propose Group Causal Counterfactual Policy Optimization to explicitly train LLMs to learn generalizable reasoning patterns. It proposes an episodic causal counterfactual reward that jointly captures (i) robustness, encouraging the answer distribution induced by a reasoning step to remain stable under counterfactual perturbations; and (ii) effectiveness, enforcing sufficient variability so that the learned reasoning strategy can transfer across questions. We then construct token-level advantages from this reward and optimize the policy, encouraging LLMs to favor reasoning patterns that are process-valid and counterfactually robust. Extensive experiments on diverse benchmarks demonstrate its advantages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。