构建多语言代码生成检测基准,助力识别生产级AI编程风险
CodeMirage: A Multi-Lingual Benchmark for Detecting AI-Generated and Paraphrased Source Code from Production-Level LLMs
- 覆盖十种主流编程语言,包含原始与改写代码
- 基于十款顶尖生产级大模型生成数据,真实反映使用场景
- 为检测器评估提供可复现的测试平台,适合安全与工具开发者
大型语言模型(LLMs)已成为现代软件开发的核心工具,生成了大量AI生成代码。尽管提升了编程效率,其滥用可能引发代码抄袭、许可证违规和不安全程序传播等风险。因此,高效检测AI生成代码至关重要。现有基准存在局限:仅覆盖少数编程语言,且依赖较弱生成模型。本文提出CodeMirage,通过三大改进克服上述问题:(1) 覆盖十种广泛使用的编程语言;(2) 包含原始代码与改写版本;(3) 涵盖来自六家主要厂商的十款先进生产级LLMs(含推理与非推理模型)生成的代码。利用CodeMirage,我们对十种代表性检测器在四种方法范式下进行评估,采用三种互补指标,在四种现实配置中报告结果。分析揭示九项关键发现,揭示当前检测器的优势与不足,并指明未来研究的关键挑战。我们认为CodeMirage为开发鲁棒、通用的AI生成代码检测器提供了严谨且实用的测试平台。
原文摘要 · Abstract (English)
Large language models (LLMs) have become integral to modern software development, producing vast amounts of AI-generated source code. While these models boost programming productivity, their misuse introduces critical risks, including code plagiarism, license violations, and the propagation of insecure programs. As a result, robust detection of AI-generated code is essential. To support the development of such detectors, a comprehensive benchmark that reflects real-world conditions is crucial. However, existing benchmarks fall short -- most cover only a limited set of programming languages and rely on less capable generative models. In this paper, we present CodeMirage, a comprehensive benchmark that addresses these limitations through three major advancements: (1) it spans ten widely used programming languages, (2) includes both original and paraphrased code samples, and (3) incorporates outputs from ten state-of-the-art production-level LLMs, including both reasoning and non-reasoning models from six major providers. Using CodeMirage, we evaluate ten representative detectors across four methodological paradigms under four realistic evaluation configurations, reporting results using three complementary metrics. Our analysis reveals nine key findings that uncover the strengths and weaknesses of current detectors, and identify critical challenges for future work. We believe CodeMirage offers a rigorous and practical testbed to advance the development of robust and generalizable AI-generated code detectors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。