首个自动评估代码生成正确性与安全性的框架，发现主流模型联合表现极低。
DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
- 构建语言无关的307个任务基准，每项含功能与安全测试
- 最强模型在三语言中联合成功率均低于15%
- 迭代式编程代理不优于直接生成，安全缺陷多集中在接口边界
大型语言模型（LLMs）及基于它们的编程代理现被用于从自然语言规范生成代码，但确保生成代码兼具功能正确性与安全性仍是挑战。本文提出DualGauge，首个完全自动化的联合评估框架，依托DualGauge-Bench基准，包含307个编程任务，每个任务均配有由同一规范衍生的功能测试与安全测试。在Python、C++和JavaScript中评估10个代表性大模型，发现功能正确性严重高估了可靠代码生成能力：即使最强模型在各语言中联合安全-功能性成功均未超过15%。常见模型因素——规模、思维链扩展、量化、指令微调和代码专业化——均无法稳定提升联合性能，表明安全且正确的代码生成并非单纯由更强编码能力产生。对三个领先编程代理系统（Codex、OpenHands、Claude Code）的评估显示，在仅基于规范的任务上，迭代式结构搭建并未带来优势。定性审计揭示失败集中于输出契约边界以及存在但不足的保护逻辑——这些模式唯有联合基准才能可靠暴露。
原文摘要 · Abstract (English)
Large language models (LLMs) and LLM-based coding agents are now used to generate code from natural-language specifications, yet ensuring such code is both functionally correct and secure remains a challenge. We present DualGauge, the first fully automated framework for jointly evaluating correctness and security of specification-only code generation, supported by DualGauge-Bench, a language-agnostic benchmark of 307 coding tasks each paired with functional and security tests derived from the same specification. Evaluating 10 representative LLMs across Python, C++, and JavaScript, we find that functional correctness substantially overestimates reliable code generation: even the strongest model remains below 15% joint security-functionality success in every language. Common model-side factors--scale, extended thinking, quantization, instruction tuning, and code specialization--do not reliably improve joint performance, suggesting secure-and-correct code generation does not simply emerge from stronger coding capability. Evaluation of 3 leading agentic coding systems (Codex, OpenHands, and Claude Code) shows that iterative scaffolding provides no advantage over direct (LLM-based) generation on specification-only tasks. A qualitative audit reveals failures concentrate at the output contract boundary and in guards that exist but are insufficient--patterns that only joint benchmarking reliably exposes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。