arXiv:2607.12273cs.SEcs.AI2026-07

用运行时行为评估代码大模型的不确定性,更准地识别高风险输出。

Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs

论文配图:Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs
图 1 · 摘自论文原文
  • 基于代码执行生成语义交互图,从运行结果判断不确定性。
  • 在8个主流模型上相关性达-0.98,显著优于传统文本方法。
  • 适合需要高可靠性的代码自动化场景,如安全审查或生成审核。

随着代码大语言模型在现代软件工程中的广泛应用,其内在的随机性带来了重大现实风险,微小错误可能导致功能、安全或安全后果。可靠的自动化要求能区分有把握的预测与随机猜测。然而,现有不确定性估计方法存在关键缺陷:白盒和灰盒技术通常无法用于闭源模型,而标准黑盒文本指标难以捕捉代码特有的脆弱性——语法变化未必带来语义差异。为此,我们提出Code-MUE,一种纯黑盒框架,通过基于执行的语义交互图测量不确定性。不同于依赖表面文本相似度的先前方法,Code-MUE将不确定性建立在可观测的运行时行为上,通过计算解空间的冯诺依曼熵来量化全局语义多样性。大规模实证研究覆盖八个先进代码LLM,结果显示Code-MUE与功能性正确性呈强负相关(斯皮尔曼相关系数最高达-0.98),显著优于词法与嵌入基线,同时在实际工作流中实现稳健的风险检测与选择性预测。

原文摘要 · Abstract (English)

As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences. Reliable automation, therefore, demands the ability to distinguish between confident, well-supported predictions and stochastic guessing. However, existing uncertainty estimation methods face a critical gap: white and grey-box techniques are often inapplicable to closed-source models, while standard "black-box" text metrics fail to capture the unique fragility of code, where syntactic variation does not always imply semantic divergence. To bridge this syntax-semantics gap, we introduce Code-MUE, a purely black-box framework that measures uncertainty through execution-based Semantic Interaction Graphs. Different from prior approaches that rely on superficial textual similarity, Code-MUE grounds uncertainty in observable runtime behavior, calculating the Von Neumann entropy of the solution space to quantify global semantic diversity. A large-scale empirical study across eight state-of-the-art LLMs demonstrates that Code-MUE achieves a strong negative correlation with functional correctness (Spearman's correlation up to -0.98), significantly outperforming lexical and embedding-based baselines while enabling robust risk detection and selective prediction in practical workflows.

代码生成不确定性大模型评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。