arXiv:2409.01382cs.SEcs.AI2024-09被引 3

对比四款大模型生成代码,发现检测效果随粒度变化极大,不能一概而论。

Automatic Detection of LLM-Generated Code: A Comparative Case Study of Contemporary Models Across Function and Class Granularities

  • 用可解释指标比较四模型在函数与类级代码上的生成差异
  • 类级别检测效果优于函数级别,现代模型在类级更易被识别
  • 检测模型需跨模型、跨粒度训练,避免依赖单一模型特征

大型语言模型(LLMs)用于代码生成存在引入漏洞的风险。现有检测方法存在两大缺陷:缺乏系统性跨模型验证和操作不透明。本文对GPT-3.5、Claude 3 Haiku、Claude Haiku 4.5和GPT-OSS四款模型进行对比研究,基于CodeSearchNet数据集中的14,485个函数和11,913个类生成对应代码。采用可解释软件度量,为每种配置训练CatBoost分类器。分析表明,粒度效应比模型差异强8.6倍,特征重叠极低,说明函数级与类级检测依赖完全不同的结构特征。发现关键的粒度相关反转:现代模型(Claude、GPT-OSS)在类级更易检测,而GPT-3.5在函数级表现异常突出。SHAP分析确认注释与代码比例是唯一通用判别特征,但其预测强度在不同模型间差异巨大,导致基于特定模型训练的检测器难以泛化。结果表明,GPT-3.5的高可检测性(AUC-ROC 0.96)不具备代表性,当代模型的AUC-ROC介于0.68至0.80之间。稳健检测必须超越单模型研究,考虑架构与粒度带来的结构指纹多样性。

原文摘要 · Abstract (English)

The adoption of Large Language Models (LLMs) for code generation risks incorporating vulnerable code into software systems. Existing detectors face two critical limitations: a lack of systematic cross-model validation and opaque "black box" operation. We address this through a comparative study of code generated by four distinct LLMs: GPT-3.5, Claude 3 Haiku, Claude Haiku 4.5, and GPT-OSS. Analyzing 14,485 Python functions and 11,913 classes from the CodeSearchNet dataset, we generated corresponding code with all four LLMs. Using interpretable software metrics, we trained CatBoost classifiers for each configuration. Our analysis reveals that granularity effects dominate model differences by a factor of 8.6, with negligible feature overlap, indicating that function-level and class-level detection rely on fundamentally disjoint structural signatures. We discover critical granularity-dependent inversions: while modern models (Claude, GPT-OSS) are more detectable at the class level, GPT-3.5 is an anomaly that uniquely excels at the function level. SHAP analysis identifies the Comment-to-Code Ratio as the sole universal discriminator. However, its predictive magnitude varies drastically across models, explaining why detectors trained on specific LLMs fail to generalize. Our findings demonstrate that GPT-3.5's exceptional detectability (AUC-ROC 0.96) is unrepresentative of contemporary models (AUC-ROC approximately between 0.68 and 0.80). Robust detection requires moving beyond single-model studies to account for substantial diversity in structural fingerprints across architectures and granularities.

代码生成检测模型粒度分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。