提升代码大模型的置信度准确性，让预测更可靠。
Multicalibration for LLM-based Code Generation
- 用多校准方法结合代码复杂度等特征优化置信度。
- 相比原始置信度提升1.03分，比基线提升0.37分。
- 适合关注模型可靠性与可解释性的研究者。
随着基于AI的代码生成广泛应用，研究人员开始关注代码大模型的校准问题——确保其置信度分数真实反映代码正确性的概率。本文研究多校准方法，能够捕捉代码问题的复杂度、长度、编程语言等额外因素。在三个函数合成基准上，使用最新代码大模型（Qwen3 Coder、GPT-OSS、DeepSeek-R1-Distill）测试四种多校准方法。结果表明，多校准可显著优于未经校准的词元似然（技能得分提升+1.03），也优于基线校准方法（提升+0.37）。通过消融实验分析了上述因素的影响，并公开包含代码生成、似然值和正确性标签的数据集，以支持未来代码大模型校准研究。
原文摘要 · Abstract (English)
As AI-based code generation becomes widespread, researchers are investigating the calibration of code LLMs - ensuring their confidence scores faithfully represent the true likelihood of code correctness. To do so, we investigate multicalibration, which can capture additional factors about a coding problem, such as complexity, code length, or programming language used. We study four multicalibration approaches on three function synthesis benchmarks, using latest-generation code LLMs (Qwen3 Coder, GPT-OSS, DeepSeek-R1-Distill). Our results demonstrate that multicalibration can yield distinct improvements over both uncalibrated token likelihoods (+1.03 in skill score) and baseline calibrations (+0.37 in skill score). We study the influence of the aforementioned factors in ablations, and make our dataset (consisting of code generations, likelihoods, and correctness labels) available for future research on code LLM calibration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。