用配对查询精准筛选大模型生成的代码，提升正确率。
ExPairT-LLM: Exact Learning for LLM Code Selection by Pairwise Queries
- 通过配对成员与等价性查询，让大模型判断代码优劣。
- 在四个数据集上平均提升13.0%，最高达27.1%成功率。
- 适合需要高精度代码生成的复杂推理场景。
尽管大语言模型（LLM）取得进展，代码生成任务仍具挑战性。现有代码选择算法常因误判非等价代码或过度依赖模型自身判断而失效。本文提出ExPairT-LLM，一种基于配对查询的精确学习算法：通过向LLM提问两种新类型问题——配对成员查询与配对等价查询，以锦标赛方式筛选最优程序。该方法更易被模型理解，且对部分错误具有鲁棒性。在四个主流代码数据集上的评估显示，其pass@1（成功率）平均优于当前最优算法13.0%，最高提升27.1%；同时使执行复杂推理的LLM pass@1提升24.0%。
原文摘要 · Abstract (English)
Despite recent advances in LLMs, the task of code generation is still challenging. To cope, code selection algorithms select the best program from multiple programs generated by an LLM. However, existing algorithms can fail to identify the correct program, either because they can misidentify nonequivalent programs or because they rely on an LLM and assume it always correctly determines the output for every input. We present ExPairT-LLM, an exact learning algorithm for code selection that selects a program by posing to an LLM oracle two new types of queries: pairwise membership and pairwise equivalence. These queries are simpler for LLMs and enable ExPairT-LLM to identify the correct program through a tournament, which is robust to some LLM mistakes. We evaluate ExPairT-LLM on four popular code datasets. Its pass@1 (success rate) outperforms the state-of-the-art code selection algorithm on average by +13.0% and up to +27.1%. It also improves the pass@1 of LLMs performing complex reasoning by +24.0%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。