让大模型的自信与推理过程对得上，避免盲目自信。
CoRA: Confidence-Rationale Alignment for Reliable Chain-of-Thought Reasoning
- 用强化学习统一优化答案正确性、置信度和推理质量。
- 在三个数据集上降低26.51%的自信与推理不一致错误。
- 适合需要可信推理的医疗、数学等专业问答场景。
链式思维（CoT）推理可提升大模型表现，但高置信度答案可能因推理过程看似合理却缺乏支撑而误导。本文研究置信度与推理理由的一致性：模型对其答案的自信是否由生成的推理过程合理支持。提出基于GRPO的强化学习框架，联合奖励答案正确性、承诺答案概率及基于评分标准的推理支持度，评分标准评估推理的依据性、连贯性、任务匹配度和与答案的关联性，且不向评判者透露正确答案。在MedQA、MathQA和OpenBookQA三个数据集上，使用三种开源大模型，相比未调优检查点、监督微调及仅奖励正确性的GRPO方法，本方法将置信度-推理一致性误差降低最多达26.51%，同时保持良好准确率并常改善校准效果。结果表明，可靠的链式思维不仅需要自信的答案，还需实质性支持的答案推理。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) reasoning can improve LLM performance, but high answer confidence may be misleading when the accompanying CoT rationale is plausible yet incomplete or poorly supported. We study confidence--rationale alignment: whether a model's confidence in its committed answer is justified by its generated rationale. We introduce a GRPO-based reinforcement learning framework that jointly rewards answer correctness, committed-answer probability, and rubric-based rationale support, where the rubric assesses grounding, coherence, task match, and connection to the selected answer without revealing the gold answer to the judge. Across MedQA, MathQA, and OpenBookQA using three open-weight LLMs, our method reduces the confidence--rationale alignment error by up to 26.51% compared with untuned checkpoints, SFT, and correctness-only GRPO, while maintaining competitive accuracy and often improving calibration. These results show that reliable CoT reasoning requires not only confident answers, but rationales that substantively support them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。