用大模型零样本识别代码作者,准确率超65%。
I Can Find You in Seconds! Leveraging Large Language Models for Code Authorship Attribution
- 用大模型零样本判断代码是否同源,无需训练
- 少样本下准确率达68.7%,支持跨语言识别
- 提出锦标赛方法,可扩展至上千作者
源代码作者归属在软件取证、抄袭检测和保护补丁完整性中具有重要意义。现有技术多依赖监督学习,因需大量标注数据,在不同编程语言和编码风格间泛化能力差。受大语言模型(LLM)在自然语言作者分析中无任务调优即表现优异的启发,本文探索使用LLM进行代码作者归属。实验表明,先进LLM可在零样本提示下成功识别跨语言代码作者,实现0.78的马修斯相关系数(MCC);通过少样本学习,仅用少量参考代码即可达到0.77的MCC。此外,模型对误导性攻击具有一定鲁棒性。然而,直接提示法在作者数量庞大时受限于输入长度。为此,提出锦标赛式方法,在C++(500作者,26,355样本)和Java(686作者,55,267样本)数据集上,仅需每作者一个参考样本,最高分类准确率达65%(C++)和68.7%(Java)。这些结果为网络安全与软件工程中的代码作者识别开辟新路径。
原文摘要 · Abstract (English)
Source code authorship attribution is important in software forensics, plagiarism detection, and protecting software patch integrity. Existing techniques often rely on supervised machine learning, which struggles with generalization across different programming languages and coding styles due to the need for large labeled datasets. Inspired by recent advances in natural language authorship analysis using large language models (LLMs), which have shown exceptional performance without task-specific tuning, this paper explores the use of LLMs for source code authorship attribution. We present a comprehensive study demonstrating that state-of-the-art LLMs can successfully attribute source code authorship across different languages. LLMs can determine whether two code snippets are written by the same author with zero-shot prompting, achieving a Matthews Correlation Coefficient (MCC) of 0.78, and can attribute code authorship from a small set of reference code snippets via few-shot learning, achieving MCC of 0.77. Additionally, LLMs show some adversarial robustness against misattribution attacks. Despite these capabilities, we found that naive prompting of LLMs does not scale well with a large number of authors due to input token limitations. To address this, we propose a tournament-style approach for large-scale attribution. Evaluating this approach on datasets of C++ (500 authors, 26,355 samples) and Java (686 authors, 55,267 samples) code from GitHub, we achieve classification accuracy of up to 65% for C++ and 68.7% for Java using only one reference per author. These results open new possibilities for applying LLMs to code authorship attribution in cybersecurity and software engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。