arXiv:2510.22614cs.SEcs.AI2025-10

测试大模型在开发环境中的校准效果,发现通用校准无效,个性化需足够数据。

Does In-IDE Calibration of Large Language Models work at Scale?

  • 用普拉特缩放法构建可扩展校准框架,适配多种开源模型与数据集。
  • 分析超2400万次真实开发者交互,发现通用校准未提升模型置信度可靠性。
  • 开发者更倾向用颜色编码而非数字显示代码生成可信度,适合集成到编辑器中。

将大语言模型引入集成开发环境正重塑软件工程,但人工智能生成代码的可用性与可靠性面临挑战。事后校准内部模型置信度旨在使概率与可接受性度量对齐。已有研究认为校准可改善对齐,但缺乏大规模证据。本文研究在开发环境中应用代码模型校准的可行性,涵盖两方面:(1)实现置信度校准的技术方法及提升代码生成模型可靠性;(2)有效向开发者传达可靠性信号的人因设计原则。首先,我们提出一个可扩展、灵活的校准框架,可使用任意数据集为开源模型获取校准权重,并评估校准器是否提升模型置信度与开发者接受行为之间的对齐。通过对跨多种编程语言的超过2400万次真实开发者交互的大规模分析,发现基于普拉特缩放的一般性事后校准模型并未平均提升置信度信号的可靠性。同时发现,动态个性化校准虽可能有效,但其效果高度依赖用户交互数据量。其次,通过包含3位专家设计师和153名专业开发者的多阶段设计研究,结合情景设计、半结构化访谈与问卷验证,揭示出开发者明确偏好在编辑器内通过非数值的彩色标识呈现可靠性信号。

原文摘要 · Abstract (English)

The introduction of large language models into integrated development environments (IDEs) is revolutionizing software engineering, yet it poses challenges to the usefulness and reliability of Artificial Intelligence-generated code. Post-hoc calibration of internal model confidences aims to align probabilities with an acceptability measure. Prior work suggests calibration can improve alignment, but at-scale evidence is limited. In this work, we investigate the feasibility of applying calibration of code models to an in-IDE context. We study two aspects of the problem: (1) the technical method for implementing confidence calibration and improving the reliability of code generation models, and (2) the human-centered design principles for effectively communicating reliability signal to developers. First, we develop a scalable and flexible calibration framework which can be used to obtain calibration weights for open-source models using any dataset, and evaluate whether calibrators improve the alignment between model confidence and developer acceptance behavior. Through a large-scale analysis of over 24 million real-world developer interactions across multiple programming languages, we find that a general, post-hoc calibration model based on Platt-scaling does not, on average, improve the reliability of model confidence signals. We also find that while dynamically personalizing calibration to individual users can be effective, its effectiveness is highly dependent on the volume of user interaction data. Second, we conduct a multi-phase design study with 3 expert designers and 153 professional developers, combining scenario-based design, semi-structured interviews, and survey validation, revealing a clear preference for presenting reliability signals via non-numerical, color-coded indicators within the in-editor code generation workflow.

大模型开发环境校准人机交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。