用LLM找代码等价性的反例，发现测试用例遗漏了18%的错误
Disproving Program Equivalence with LLMs
- 通过执行反馈让LLM主动搜索代码等价性反例
- 在基准测试中发现18%原被认为等价的代码实际不等价
- 可用于提升代码生成的一致性，让模型更靠谱
为评估大语言模型在代码任务上的表现，研究通常依赖人工编写的单元测试基准。然而这些测试常因遗漏边界情况和实现细节而失效。本文提出ProbeGen，一种白盒方法，可对多个可执行代码片段进行等价性验证，并主动搜索反例。由于代码语义理解需要深度推理，我们证明具备执行反馈的LLM在此任务上表现优异。在常见代码合成基准中，ProbeGen发现了18%被基准测试误判为等价于真实答案的样本。此外，利用ProbeGen可对LLM生成结果进行语义聚类，统一语法不同但语义相同的代码，使pass@1指标提升10%。
原文摘要 · Abstract (English)
To evaluate large language models (LLMs) for code, research has used manually created unit test-based benchmarks. However, these tests are often inadequate, missing corner cases and other implementation-specific oddities. This work introduces ProbeGen, a whitebox method that takes two or more executable pieces of code and searches for counterexamples to their equivalence. Comparing code semantics requires a deep understanding of code. We demonstrate that LLMs with execution feedback perform well at this task. In a common code synthesis benchmark, ProbeGen disproves 18% of samples considered equivalent to the ground truth by the benchmark-provided unit tests. Additionally, using ProbeGen, we can semantically cluster LLM samples for semantic self-consistency, improving pass@1 by 10% by unifying syntactically distinct but semantically similar samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。