研究大模型生成代码时的安全自信程度,发现普遍高估自身安全性。
An Empirical Study of Security Calibration in Large Language Models for Code

- 通过多温度设置测试三个大模型在两类安全任务中的自信度表现。
- 模型对功能正确性的判断比安全问题更不靠谱,且修复漏洞效果有限。
- 提升安全校准的策略在真实项目中反而恶化表现,增加误信风险。
大型语言模型正快速改变软件开发,但其在安全关键场景中的应用引发核心问题:模型是否知道自身生成的代码是否存在安全隐患?这一特性称为校准性,即模型自信程度与输出真实正确性的匹配度。本文首次开展大规模实证研究,评估 GPT-4o-mini、Gemini-2.0-Flash 与 Qwen3-Coder-Next 在多个温度设置下于两类互补基准上的安全校准表现:自包含安全任务与跨语言仓库级上下文。结果表明,模型普遍存在过度自信现象。功能校准表现持续劣于安全校准,可能因功能正确性依赖复杂执行行为。我们还考察了基于校准引导的自动化修复能否缓解漏洞,发现改进有限且常引入功能退化。此外,研究了降低‘误信’(模型对有漏洞代码赋予高置信)的不同缓解策略,结果显示尽管架构门控在受控基准上改善校准,但在真实仓库级场景中校准性能下降,反而增加了高置信度漏洞输出的风险。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are rapidly transforming software development, yet their use in security-critical contexts raises a key question: do models know when their generated code is insecure? This property, known as calibration, measures whether a model's confidence aligns with the true correctness of its outputs. We present the first large-scale empirical study of security calibration in LLM-generated code. We evaluate GPT-4o-mini, Gemini-2.0-Flash, and Qwen3-Coder-Next across multiple temperature settings on two complementary benchmarks: self-contained security tasks and multi-language repository-level contexts. Our results suggest that overconfidence is prevalent across the evaluated LLMs. Functional calibration is consistently worse than security calibration, suggesting that models estimate security outcomes more reliably than functional correctness, potentially because functional correctness depends on complex execution behavior. We also examine whether calibration-guided automated repair can help remediate vulnerabilities in LLM-generated code, finding only limited improvements while frequently introducing functional regressions. Moreover, we study different mitigation strategies for reducing False Trust, where models assign high confidence to vulnerable code. The results show that although architectural gating improves calibration on controlled benchmarks, calibration deteriorates in realistic repository-level settings, increasing the risk of high-confidence vulnerable outputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。