模仿人类思维分步解题,让大模型更可靠地推理
From Meta-Thought to Execution: Cognitively Aligned Post-Training for Generalizable and Reliable LLM Reasoning
- 先学通用解题思路,再优化具体执行
- 在分布内/外测试中分别提升2.10%和3.86%
- 适合需要稳定推理的复杂任务场景
当前大模型后训练方法通过监督微调(SFT)结合基于结果的强化学习(RL)优化完整推理轨迹。但这种方法与人类真实解题方式存在根本差异:人类会先掌握跨问题的抽象策略(元知识),再将其适配到具体任务。现有方法将完整轨迹视为基本单元,导致抽象策略与具体执行纠缠。为此,我们提出一种类人认知框架——链式元思维(CoMT),在监督学习中聚焦抽象推理模式,不包含具体执行,以获取可泛化的策略;再通过置信度校准强化学习(CCRL)优化任务适配,基于中间步骤的置信度奖励防止过度自信错误传播,提升执行可靠性。在四个模型、十个基准上的实验表明,该方法在分布内和分布外分别取得2.10%和3.86%的性能提升,且对教师模型选择、优化方法及符号扰动具有高度鲁棒性。
原文摘要 · Abstract (English)
Current LLM post-training methods optimize complete reasoning trajectories through Supervised Fine-Tuning (SFT) followed by outcome-based Reinforcement Learning (RL). While effective, a closer examination reveals a fundamental gap: this approach does not align with how humans actually solve problems. Human cognition naturally decomposes problem-solving into two distinct stages: first acquiring abstract strategies (i.e., meta-knowledge) that generalize across problems, then adapting them to specific instances. In contrast, by treating complete trajectories as basic units, current methods are inherently problem-centric, entangling abstract strategies with problem-specific execution. To address this misalignment, we propose a cognitively-inspired framework that explicitly mirrors the two-stage human cognitive process. Specifically, Chain-of-Meta-Thought CoMT focuses supervised learning on abstract reasoning patterns without specific executions, enabling acquisition of generalizable strategies. Confidence-Calibrated Reinforcement Learning (CCRL) then optimizes task adaptation via confidence-aware rewards on intermediate steps, preventing overconfident errors from cascading and improving execution reliability. Experiments across four models and ten benchmarks show 2.10% and 3.86% improvements in-distribution and out-of-distribution respectively over standard methods, while remaining highly robust to variations in teacher model selection, optimization methods, and symbolic perturbations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。