新基准DeepBench评估大模型生成完整深度学习代码的能力。
Deep-Bench: Deep Learning Benchmark Dataset for Code Generation
- 按数据类型、任务和流程阶段分类，覆盖完整DL开发链
- GPT-4o在新基准上准确率仅31%，远低于旧基准的60%
- 揭示大模型生成深度学习代码时的独特错误模式
深度学习（DL）已革新计算机视觉、自然语言处理等领域，但其复杂的工作流使系统开发困难。大语言模型（LLMs）如GPT、Claude、Llama、Mistral等为辅助生成DL代码提供了新可能。然而，现有基准如DS-1000受限于仅关注预/后处理的小片段代码，缺乏对完整DL流程（包括预处理、模型构建、训练等阶段）及多种输入数据类型（表格、图像、文本）的全面覆盖。为此，我们提出DeepBench，一个面向函数级深度学习代码生成的新基准数据集。该数据集基于三个维度：流程阶段、任务类型（分类、回归、推荐）和输入数据类型。实验显示，GPT-4o在DeepBench上的准确率为31%，显著低于其在DS-1000上的60%；其他模型表现亦呈类似趋势（Claude：28% vs. 54%；LLaMA：21% vs. 41%；Mistral：15% vs. 20%）。分析还发现不同类别间性能差异明显，阶段间最大差值达7%，任务间达37%。我们进一步构建了生成代码中常见问题与缺陷的分类体系，揭示了大模型在生成深度学习代码时面临的独特挑战。
原文摘要 · Abstract (English)
Deep learning (DL) has revolutionized areas such as computer vision, natural language processing, and more. However, developing DL systems is challenging due to the complexity of DL workflows. Large Language Models (LLMs), such as GPT, Claude, Llama, Mistral, etc., have emerged as promising tools to assist in DL code generation, offering potential solutions to these challenges. Despite this, existing benchmarks such as DS-1000 are limited, as they primarily focus on small DL code snippets related to pre/post-processing tasks and lack a comprehensive coverage of the full DL pipeline, including different DL phases and input data types. To address this, we introduce DeepBench, a novel benchmark dataset designed for function-level DL code generation. DeepBench categorizes DL problems based on three key aspects: phases such as pre-processing, model construction, and training; tasks, including classification, regression, and recommendation; and input data types such as tabular, image, and text. GPT-4o -- the state-of-the-art LLM -- achieved 31% accuracy on DeepBench, significantly lower than its 60% on DS-1000. We observed similar difficulty for other LLMs (e.g., 28% vs. 54% for Claude, 21% vs. 41% for LLaMA, and 15% vs. 20% for Mistral). This result underscores DeepBench's greater complexity. We also construct a taxonomy of issues and bugs found in LLM-generated DL code, which highlights the distinct challenges that LLMs face when generating DL code compared to general code. Furthermore, our analysis also reveals substantial performance variations across categories, with differences of up to 7% among phases and 37% among tasks. These disparities suggest that DeepBench offers valuable insights into the LLMs' performance and areas for potential improvement in the DL domain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。