用函数等价性检测大模型生成代码的错误，比传统方法更准。
Functional Entropy: Predicting Functional Correctness in LLM-Generated Code with Uncertainty Quantification
- 用大模型评估代码功能等价性，替代依赖自然语言推理的方法。
- 在15个组合中11次获得最高准确率，且结果校准性更好。
- 适合关注代码生成质量与可靠性研究者使用。
大型语言模型在代码生成方面表现优异，但常产生功能错误的代码。不确定性量化（UQ）方法在自然语言生成中被用于检测幻觉，但在代码生成任务中的有效性尚未充分探索。本文系统评估了多种UQ技术在三种编程语言、五种大模型及超过1,700个问题上的迁移效果。发现部分基于词元概率的方法无需修改即可有效泛化，而依赖自然语言推理（NLI）的采样方法因无法区分功能不同的代码，导致多数响应聚集为单一语义簇而失效。为此，我们提出「功能等价方法」，以大模型驱动的功能等价性评估取代NLI，包括一种针对代码的函数熵（functional entropy），作为语义熵的代码版。该方法在15组模型-基准组合中，有11组取得最高AUROC，并在多数设置下表现最佳校准性，显著优于基于NLI的方法及其他评估方法。
原文摘要 · Abstract (English)
Large language models have shown impressive capabilities in code generation, yet they often produce functionally incorrect code. Uncertainty quantification (UQ) methods have emerged as a promising approach for detecting hallucinations in natural language generation, but their effectiveness for code generation tasks remains underexplored. We systematically evaluate how UQ techniques transfer to code generation across three programming languages, five LLMs, and over 1,700 problems. We find that some token-probability-based methods generalize effectively without modification, while sampling-based methods relying on natural language inference (NLI) fail because NLI models cannot distinguish functionally different code, causing most responses to collapse into a single semantic cluster. To address this, we introduce \emph{functional equivalence methods}, a family of code-specific methods that replace NLI-based semantic equivalence with an LLM-based functional equivalence assessment, including functional entropy, a code-specific analog of semantic entropy. Functional equivalence methods achieve top AUROC in 11 out of 15 model-benchmark combinations and the best calibration across most settings, consistently outperforming both NLI-based counterparts and all other methods evaluated.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。