通过自信度对齐提升大模型推理可靠性，无需额外奖励模型。
Confidence-Aware Alignment Makes Reasoning LLMs More Reliable
- 用迭代偏好优化让每步推理的自信度与逻辑正确性对齐。
- 推理时动态剪枝不确定分支，延迟仅增加O(V)。
- 适用于多种模型，提升准确率与效率，适合追求可靠推理的场景。
大型推理模型常通过有缺陷的中间步骤得出正确答案，导致最终准确率与推理可靠性之间存在差距。现有对齐方法依赖外部验证器或大量采样，难以扩展。本文提出CASPO（自信度感知的逐步偏好优化）框架，通过迭代直接偏好优化，在不训练独立奖励模型的前提下，实现令牌级自信度与步骤级逻辑正确性的对齐。推理阶段引入自信度感知思维（CaT），利用校准后的自信度动态剪枝不确定推理分支，计算开销仅增加O(V)。在十个基准测试及多个模型家族上的实验表明，CASPO持续提升推理可靠性与推理效率。CASPO可扩展至Qwen3-8B-Base，在AIME'24和AIME'25上超越树搜索基线，且无需使用奖励模型数据。我们还发布了带自信度标注的逐步数据集，以支持推理可靠性细粒度分析。代码已开源：https://github.com/Thecommonirin/CASPO。
原文摘要 · Abstract (English)
Large reasoning models often reach correct answers through flawed intermediate steps, creating a gap between final accuracy and reasoning reliability. Existing alignment strategies address this with external verifiers or massive sampling, limiting scalability. In this work, we introduce CASPO (Confidence-Aware Step-wise Preference Optimization), a framework that aligns token-level confidence with step-wise logical correctness through iterative Direct Preference Optimization, without training a separate reward model. During inference, we propose Confidence-aware Thought (CaT), which leverages this calibrated confidence to dynamically prune uncertain reasoning branches with negligible O(V) latency. Experiments across ten benchmarks and multiple model families show that CASPO consistently improves reasoning reliability and inference efficiency. CASPO scales to Qwen3-8B-Base and surpasses tree-search baselines on AIME'24 and AIME'25 without using reward-model data. We also release a step-wise dataset with confidence annotations to support fine-grained analysis of reasoning reliability. Code is available at https://github.com/Thecommonirin/CASPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。