arXiv:2601.06423cs.AI2026-01被引 2

测试大模型推理一致性时,发现提升准确率未必提升真实推理能力。

Does Inference Scaling Improve Reasoning Faithfulness? A Multi-Model Analysis of Self-Consistency Tradeoffs

  • 多路径生成+多数投票提升准确率,但效果因模型而异。
  • 部分模型准确率下降,但推理可信度反而飙升,矛盾现象显著。
  • 建议实测自一致性对特定模型的影响,避免盲目部署。

自一致性已成为提升大模型推理任务准确率的常用方法:生成多条推理路径后通过多数投票选择答案。尽管该方法能稳定提高准确率,但其是否真正改善了推理质量仍不明确。本文首次系统研究推理规模扩展是否提升推理可信度。我们在100道GSM8K数学题上对四个前沿模型(GPT-5.2、Claude Opus 4.5、Gemini-3-flash-preview、DeepSeek-v3.2)进行实证分析,采用自助法置信区间、配对McNemar检验和Cohen's d效应量量化结果。结果显示:GPT-5.2准确率从78%升至90%,可信度略有下降(0.540→0.510);Claude Opus 4.5准确率反降至74.3%,可信度却从0.270飙升至0.891;DeepSeek-v3.2已达98%准确率,可信度仅从0.440微增至0.541;Gemini-3-flash准确率从81%升至86%,可信度轻微下降(0.260→0.212)。难度分析表明,GPT-5.2解决82%难题,仅错13%易题;而Claude错23%易题,解释其准确率下降。结论:自一致性非普适有效,需针对具体模型测试。

原文摘要 · Abstract (English)

Self-consistency has emerged as a popular technique for improving large language model accuracy on reasoning tasks. The approach is straightforward: generate multiple reasoning paths and select the most common answer through majority voting. While this reliably boosts accuracy, it remains unclear whether these gains reflect genuine improvements in reasoning quality. We investigate a fundamental question that has not been studied before: does inference scaling improve reasoning faithfulness? We conduct a comprehensive empirical study across four frontier models (GPT-5.2, Claude Opus 4.5, Gemini-3-flash-preview, and DeepSeek-v3.2) on 100 GSM8K mathematical reasoning problems. Our analysis employs bootstrap confidence intervals, McNemar's tests for paired comparisons, and Cohen's d effect sizes to quantify the effects rigorously. The results reveal striking differences across models that challenge common assumptions about self-consistency. GPT-5.2 shows the expected pattern: accuracy improves from 78% to 90% at N=5, with faithfulness remaining relatively stable (0.540 to 0.510). Claude Opus 4.5 tells a completely different story. Its accuracy actually drops from 78% to 74.3% while faithfulness jumps dramatically from 0.270 to 0.891 at N=5. DeepSeek-v3.2, already at 98% accuracy, shows ceiling effects with modest faithfulness gains (0.440 to 0.541). Gemini-3-flash improves from 81% to 86% accuracy with a slight faithfulness decrease (0.260 to 0.212). Problem difficulty analysis reveals that GPT-5.2 solves 82% of hard problems while breaking only 13% of easy ones. Claude, in contrast, breaks 23% of easy problems, explaining its accuracy decrease. These findings matter for practitioners: self-consistency is not universally beneficial, and teams should test their specific models before deployment. We release our code and provide practical recommendations for navigating these tradeoffs.

大模型推理可信度自一致性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。