arXiv:2511.02197cs.SEcs.AI2025-11被引 2

提升大模型代码推理可信度,实测最佳方案组合可让判断更准

Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMs

  • 设计代码推理信心评估与优化框架,结合提示词重审与数学校准
  • 混合策略使信心可靠性指标最高提升15.084分,优于单一方法
  • 适合关注大模型代码生成可信度的开发者与研究者

随着大语言模型在代码智能领域的广泛应用,其在代码推理任务中输出的可靠性与可控性日益受到关注。信心估计为此提供了有效且便捷的评估途径。本文针对代码推理任务,提出一个信心分析与增强框架,对主流大模型在不同任务中的信心可靠性进行了全面实证研究,并评估了提示优化与数学校准(如Platt Scaling)等技术的有效性。实验表明,DeepSeek-Reasoner在各类任务中表现最优,其ECE、Brier Score与性能得分分别领先其他模型最多0.680、0.636和13.652。结合重审提示策略与Platt Scaling的混合方法,三项指标相比原始性能最高提升0.541、0.628与15.084。结果表明具备推理能力的模型信心可靠性更强,且混合策略是提升各类模型信心可靠性的最有效手段。同时,研究揭示了任务复杂度、模型规模及策略选择对信心表现的影响,指出当前大模型在复杂推理任务中的信心仍有显著提升空间。本研究为大模型辅助软件工程中的信心应用提供了理论基础与技术参考,也为未来信心机制的优化与工程部署指明方向。

原文摘要 · Abstract (English)

With the widespread application of large language models (LLMs) in the field of code intelligence, increasing attention has been paid to the reliability and controllability of their outputs in code reasoning tasks. Confidence estimation serves as an effective and convenient approach for evaluating these aspects. This paper proposes a confidence analysis and enhancement framework for LLMs tailored to code reasoning tasks. We conduct a comprehensive empirical study on the confidence reliability of mainstream LLMs across different tasks, and further evaluate the effectiveness of techniques such as prompt strategy optimisation and mathematical calibration (e.g., Platt Scaling) in improving confidence reliability. Our results show that DeepSeek-Reasoner achieves the best performance across various tasks, outperforming other models by up to $0.680$, $0.636$, and $13.652$ in terms of ECE, Brier Score, and Performance Score, respectively. The hybrid strategy combining the reassess prompt strategy and Platt Scaling achieves improvements of up to $0.541$, $0.628$, and $15.084$ over the original performance in the aforementioned three metrics. These results indicate that models with reasoning capabilities demonstrate superior confidence reliability, and that the hybrid strategy is the most effective in enhancing the confidence reliability of various models. Meanwhile, we elucidate the impact of different task complexities, model scales, and strategies on confidence performance, and highlight that the confidence of current LLMs in complex reasoning tasks still has considerable room for improvement. This study not only provides a research foundation and technical reference for the application of confidence in LLM-assisted software engineering, but also points the way for future optimisation and engineering deployment of confidence mechanisms.

代码推理信心估计大模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。