构建首个混合代码作者检测数据集,助力识别真实场景中的人机共写代码。
HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection

- 通过开源仓库构建含人机交替编码行的数据集
- 最先进模型在行级检测上F1仅达0.48,任务极具挑战
- 适合研究代码安全、智能编程工具评估的开发者
随着大语言模型驱动的AI代码助手快速普及,产业级代码库正越来越多地融合人工与AI生成代码。为实现风险管控与生产效率分析,精准定位AI生成代码段至关重要。然而现有基准多基于学术性、类似LeetCode的题目,且假设代码片段要么全为人工编写,要么全由AI生成,难以反映实际使用中复杂多样的编写意图与风格。为此,我们提出HybridCodeAuthorship,一个包含人机交替编写行的Python代码文件基准数据集,以模拟真实场景下对AI代码助手的使用。本文首先介绍基于CodeSearchNet大规模开源仓库链接的数据构建流程,随后在行级和块级任务上测试两种前沿的AI生成代码检测算法性能。实验结果表明,该基准具有较高难度:表现最佳的AIGCode Detector模型在块级和行级检测上的最高F1分数分别为0.56和0.48。
原文摘要 · Abstract (English)
Thanks to the rapid adoption of AI code assistants powered by large language models (LLMs), industry codebases are, increasingly, a hybrid of AI- and human-authored code. For risk management and productivity analysis purposes, it is crucial to enable fine-grained location detection of AI-generated code. To develop algorithms for this task, quality benchmarks are needed to assess performance. However, existing benchmarks tend to comprise academic, LeetCode-style problems and presume a code snippet is either completely human-authored or completely AI-authored, which is not reflective of the diverse intents and styles of industry codebases utilizing AI code assistants. To fill these gaps, we introduce HybridCodeAuthorship, a novel benchmark of Python code files with interleaved human- and AI-authored lines of code to simulate authentic utilization of AI code assistants. In this paper, we first present our dataset construction pipeline, which leverages CodeSearchNet, a massive collection of links to open sourced repositories on GitHub. We then benchmark the performance of two state-of-the-art AI-generated code detection algorithms at both the line- and chunk-level. Experimental results demonstrate that HybridCodeAuthorship is a challenging benchmark with a top-scoring algorithm, AIGCode Detector, obtaining a highest F1 score of 0.48 and 0.56 on chunk-level and line-level code detection tasks, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。