arXiv:2604.06723cs.SEcs.AI2026-04

为代码修复中的大模型提供更精准的置信度校准,提升使用可靠性。

Fine-grained Approaches for Confidence Calibration of LLMs in Automated Code Revision

  • 按局部编辑粒度分别校准三种细粒度置信度分数。
  • 在14个不同规模模型上验证,校准误差显著降低。
  • 适合关注代码生成可信度的开发者与研究者。

在当前AI辅助软件工程背景下,开发者高度依赖能力强大但存在缺陷的大语言模型(LLMs)。这些模型产生错误输出的倾向会降低开发效率。为此,一种常见缓解方法是提供校准后的置信度分数,真实反映其输出正确的概率,帮助用户判断是否接受、回避错误输出,并合理预期模型能力。由于后训练的LLMs本身不具备良好校准的置信度,研究人员提出了后处理校准方法,其中全局的序列级置信度普拉特缩放(Platt-scaling)在多数生成式软件工程任务中表现有效,但在自动化代码修订(ACR)任务如程序修复、漏洞修复和代码优化中仍不可靠或未被充分探索。我们假设,传统粗粒度方法不适用于此类任务,因正确性常由局部修改决定,且误校准具有样本依赖性,因而提出细粒度置信度校准。为此,本研究将局部普拉特缩放应用于三种不同的细粒度置信度分数。通过在3项不同任务和正确性指标上,以及14种不同规模的模型上的实验,发现细粒度置信度分数在更广泛的概率区间内始终表现出更低的校准误差,且该效果在结合全局普拉特缩放时进一步增强。所提方法为获取可信赖的置信度分数提供了实用方案,使不完美的模型在代码修订任务中更可信、更高效地应用。

原文摘要 · Abstract (English)

In today's AI-assisted software engineering landscape, developers increasingly depend on LLMs that are highly capable, yet inherently imperfect. The tendency of these models to produce incorrect outputs can reduce developer productivity. To this end, a canonical mitigation method is to provide calibrated confidence scores that faithfully reflect their likelihood of correctness at the instance-level. Such information allows users to make immediate decisions regarding output acceptance, abstain error-prone outputs, and better align their expectations with the model's capabilities. Since post-trained LLMs do not inherently produce well-calibrated confidence scores, researchers have developed post-hoc calibration methods, with global Platt-scaling of sequence-level confidence scores proving effective in many generative software engineering tasks but remaining unreliable or unexplored for automated code revision (ACR) tasks such as program repair, vulnerability repair, and code refinement. We hypothesise that the coarse-grained nature of this conventional method makes it ill-suited for ACR tasks, where correctness is often determined by local edit decisions and miscalibration can be sample-dependent, thereby motivating fine-grained confidence calibration. To address this, our study proposes local Platt-scaling applied separately to three different fine-grained confidence scores. Through experiments across 3 separate tasks and correctness metrics, as well as 14 different models of various sizes, we find that fine-grained confidence scores consistently achieve lower calibration error across a broader range of probability intervals, and this effect is further amplified when global Platt-scaling is applied. Our proposed approaches offer a practical solution to eliciting well-calibrated confidence scores, enabling more trustworthy and streamlined usage of imperfect models in ACR tasks.

代码生成置信度校准大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。