无需正确代码作参考，用逻辑不连贯性评估大模型生成代码的错误概率。
Incoherence as Oracle-less Measure of Error in LLM-Based Code Generation
- 通过分析代码生成过程中的逻辑不连贯性，估算错误率。
- 在无正确参照下，可识别约三分之二的错误代码且无误报。
- 排名结果与传统测试方法高度一致，适合替代人工验证。
从自然语言任务生成代码是大语言模型（LLMs）最成功的应用之一，但生成的程序可能存在缺陷。当缺乏正确的实现或形式化规范等外部参考（即无预言机）时，能否估计生成程序出错的可能性？本文提出一种称为“不连贯性”的错误度量方法，可在无预言机条件下高效估算错误概率，并建立错误率的下界。实验表明，基于不连贯性的方法能自动识别约三分之二的错误程序，且对平均任务无误报。进一步发现，使用不连贯性评估的LLM排名，与基于预言机（pass@1）的排名具有极强一致性，说明前者可可靠替代后者。
原文摘要 · Abstract (English)
Generating code from a natural language programming task is one of the most successful applications of Large Language Models (LLMs). Yet, the generated program may be buggy. Without an oracle, such as an existing, correct implementation or a formal specification, can we somehow estimate how likely the generated program is correct? In this paper, we propose a measure of incorrectness, called *incoherence*, that can be estimated efficiently in the absence of an oracle and allows us to establish a lower bound on the error, i.e., the probability that the LLM-generated program for that specification is incorrect. In our experiments, our incoherence-based methodology can automatically identify about two-thirds of incorrect programs without reports of false positives for the average task. In fact, *an oracle-based evaluation of LLMs can be reliably replaced by an incoherence-based evaluation*. In particular, we find a very strong agreement between the ranking of LLMs by the number of programs deemed correct via an oracle (pass@1) and the ranking of LLMs by the number of programs deemed correct via incoherence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。