用多个大模型联合评审代码，提升自动编程的安全性。
Vibe Coding on Trial: Operating Characteristics of Unanimous LLM Juries
- 构建由强模型组成的无异议评审团，统一判定生成的SQL是否正确。
- 小规模评审团可将误通过率降至0.1以下，同时保留85%以上有效请求。
- 评审团构成对结果影响显著，适合高安全场景的自动化代码审核。
大型语言模型（LLMs）已具备足够编码能力，开发者可用自然语言描述意图，由工具生成代码初稿，这一流程正被GitHub Copilot、Cursor和Replit等广泛集成。然而，缺乏可靠方法判断哪些模型生成的查询可直接接受而不必人工审查。本文研究将LLM评审团应用于代码审查环节。首先在82个MySQL文本转SQL任务上，基于执行验证协议对15个开源模型进行基准测试，获得各模型性能基线。从表现最佳的6个模型中构建规模为1至6人的无异议评审团，评审团成员共同查看提示词、数据库模式及候选SQL，仅当所有成员一致认为正确时才接受。该规则符合安全优先部署原则——误通过代价高于误拒绝。我们评估了真阳性率、假阳性率与Youden J指数，并分析了每生成器对应评审团数量的影响。结果表明：单模型评判者表现不一；由强模型组成的小规模无异议评审团可在大幅降低误通过率的同时，仍通过多数优质请求；且评审团具体构成对结果有显著影响。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are now good enough at coding that developers can describe intent in plain language and let the tool produce the first code draft, a workflow increasingly built into tools like GitHub Copilot, Cursor, and Replit. What is missing is a reliable way to tell which model written queries are safe to accept without sending everything to a human. We study the application of an LLM jury to run this review step. We first benchmark 15 open models on 82 MySQL text to SQL tasks using an execution grounded protocol to get a clean baseline of which models are strong. From the six best models we build unanimous committees of sizes 1 through 6 that see the prompt, schema, and candidate SQL and accept it only when every member says it is correct. This rule matches safety first deployments where false accepts are more costly than false rejects. We measure true positive rate, false positive rate and Youden J and we also look at committees per generator. Our results show that single model judges are uneven, that small unanimous committees of strong models can cut false accepts while still passing many good queries, and that the exact committee composition matters significantly.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。