arXiv:2510.13859cs.SEcs.AI2025-10被引 6

首个评估多轮代码生成正确性与安全性的基准,发现模型表现大幅下降。

Benchmarking Correctness and Security in Multi-Turn Code Generation

  • 构建合成数据流,将单轮任务转为多轮交互序列。
  • 多轮场景下正确且安全输出率下降20%-27%。
  • 适合关注真实开发流程中AI代码质量的研究者。

基于大语言模型的编程助手已显著提升开发效率。现有评测多限于单轮任务,无法反映真实开发中的迭代特性。本文提出MT-Sec,首个系统评估多轮代码生成中正确性与安全性的基准。通过合成数据管道,将原有单轮任务转化为语义对齐的多轮交互序列,保留原测试集的同时模拟真实开发复杂度。评估32个开源与闭源模型及三种代理框架,在多轮场景下,“正确且安全”的输出率相比单轮下降20%-27%,即使顶尖模型也未幸免。此外,针对多轮代码差异生成这一尚未探索但实用的场景,模型表现更差,功能错误和安全隐患增多。尽管代理框架在单轮中提升性能,但在多轮评测中效果有限。研究强调需建立兼顾正确性与安全性的多轮真实工作流评测基准。

原文摘要 · Abstract (English)

AI coding assistants powered by large language models (LLMs) have transformed software development, significantly boosting productivity. While existing benchmarks evaluate the correctness and security of LLM-generated code, they are typically limited to single-turn tasks that do not reflect the iterative nature of real-world development. We introduce MT-Sec, the first benchmark to systematically evaluate both correctness and security in multi-turn coding scenarios. We construct this using a synthetic data pipeline that transforms existing single-turn tasks into semantically aligned multi-turn interaction sequences, allowing reuse of original test suites while modeling the complexity of real-world coding processes. We evaluate 32 open- and closed-source models, and three agent-scaffolding on MT-Sec and observe a consistent 20-27% drop in "correct and secure" outputs from single-turn to multi-turn settings -- even among state-of-the-art models. Beyond full-program generation, we also evaluate models on multi-turn code-diff generation -- an unexplored yet practically relevant setting -- and find that models perform worse here, with increased rates of functionally incorrect and insecure outputs. Finally, we find that while agent scaffoldings boost single-turn code generation performance, they are not quite as effective in multi-turn evaluations. Together, these findings highlight the need for benchmarks that jointly evaluate correctness and security in multi-turn, real-world coding workflows.

代码生成多轮对话安全性评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。