对比GPT-4与DeepSeek-V3检测代码异味的性能与成本
Benchmarking LLM for Code Smells Detection: OpenAI GPT-4.0 vs DeepSeek-V3
- 构建多语言代码异味数据集，系统评估大模型检测能力
- GPT-4在整体F1上优于DeepSeek-V3，但成本更高
- 适合关注性价比或需跨语言检测的开发者参考
确定最有效的大型语言模型用于代码异味检测是一项复杂挑战。本研究提出一种结构化方法与评估矩阵，利用经过一致标注的代码样本数据集进行评测。该数据集涵盖Java、Python、JavaScript和C++四种主流编程语言，支持跨语言比较。我们对OpenAI GPT-4.0与DeepSeek-V3两个前沿大模型进行了基准测试，采用精确率、召回率和F1分数作为评估指标。分析覆盖整体表现、类别级别及具体异味类型三个层面。此外，还比较了GPT-4.0基于令牌的检测方式与DeepSeek-V3的模式匹配技术的成本效益，并与传统静态分析工具（如SonarQube）进行成本对比。研究结果为实践者选择高效且经济的自动化代码异味检测方案提供了重要指导。
原文摘要 · Abstract (English)
Determining the most effective Large Language Model for code smell detection presents a complex challenge. This study introduces a structured methodology and evaluation matrix to tackle this issue, leveraging a curated dataset of code samples consistently annotated with known smells. The dataset spans four prominent programming languages Java, Python, JavaScript, and C++; allowing for cross language comparison. We benchmark two state of the art LLMs, OpenAI GPT 4.0 and DeepSeek-V3, using precision, recall, and F1 score as evaluation metrics. Our analysis covers three levels of detail: overall performance, category level performance, and individual code smell type performance. Additionally, we explore cost effectiveness by comparing the token based detection approach of GPT 4.0 with the pattern-matching techniques employed by DeepSeek V3. The study also includes a cost analysis relative to traditional static analysis tools such as SonarQube. The findings offer valuable guidance for practitioners in selecting an efficient, cost effective solution for automated code smell detection
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。