分两步评估代码正确性，先理解再审计，提升模型判断准确性。
Code Comprehension then Auditing for Unsupervised LLM Evaluation
- 先生成代码功能的自然语言解释，再据此评估任务对齐度。
- 在多个数据集上最高提升68%的F1分数和20%准确率。
- 适合需要无监督代码评估的研究者和开发者使用。
无需参考实现或单元测试，大语言模型可无监督地评估代码正确性，但现有方法直接基于完整代码进行判断，导致行为理解和正确性评估混淆，引发误判。为此，我们提出CoCoA框架，分两步进行：首先理解代码功能并生成自然语言说明，再基于该说明评估任务对齐程度。通过顺序采样理解与评估，提升了程序行为推断质量，使模型更专注于行为一致性而非实现细节。在多个数据集、编程语言和模型上，CoCoA相比最佳基线，F1分数最高提升68%，准确率最高提升20%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) for unsupervised code correctness evaluation have recently gained attention because they can judge if code runs as intended without requiring reference implementations or unit tests, which may be unavailable, sparse, or unreliable. However, most prior approaches condition LLM evaluators directly on the full code implementation, forcing the model to jointly infer program behavior and evaluate correctness in a single step. This entanglement leads to misinterpretations of code behavior and unreliable judgments. To mitigate this issue, we introduce CoCoA, an unsupervised Code Comprehension then Auditing framework that first comprehends functionality to generate a natural-language explanation. Then it evaluates task alignment based on this explanation. By sequentially sampling comprehension before evaluation, CoCoA improves the quality of inferred program behavior and enables the evaluator to focus on behavioral alignment rather than raw implementation details. Across multiple datasets, programming languages, and models, CoCoA achieves up to $68\%$ increased F1 score and up to $20\%$ increased accuracy over the best-performing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。