arXiv:2412.18573cs.SEcs.AI2024-12被引 17

测试12个领域代码生成能力,发现顶尖模型在特定领域未必强。

Top General Performance = Top Domain Performance? DomainCodeBench: A Multi-domain Code Generation Benchmark

  • 构建12领域跨语言基准,含2400个人工验证任务。
  • 顶级通用模型在特定领域表现不一,存在性能脱节现象。
  • 注入领域知识可提升38.17%效果,适合工业级代码应用者。

随着大语言模型(LLMs)的快速发展,研究普遍聚焦于通用领域代码生成能力,却忽视了真实应用场景中特定领域的表现。这引发核心问题:通用编码能力能否可靠代表特定领域能力?本文提出DomainCodeBench,一个涵盖12个软件应用领域和15种编程语言的多领域代码生成基准。该基准包含2,400个经人工验证的任务,具备真实代码上下文、人工标注文档字符串及细粒度依赖信息,以覆盖领域特有挑战。通过主题挖掘确定主流领域,基于各领域常用框架与平台构建任务。在十款主流LLM上进行实验,发现:(1) 性能解耦:顶尖通用模型在特定领域并不总是领先;(2) 领域知识缺失与第三方库误用是主要失败原因;(3) 在提示中加入领域知识可使性能提升约38.17%,为优化提供实证支持。完整复现包(含数据集、代码与结果)已开源至https://github.com/DeepSoftwareAnalytics/DomainCodeBench。

原文摘要 · Abstract (English)

With the rapid advancement of large language models (LLMs), extensive research has been conducted to investigate the code generation capabilities of LLMs. However, existing efforts primarily focus on general-domain tasks, leaving LLMs' code generation performance in real-world application domains underexplored. This raises a critical question: can a model's general-domain coding ability reliably represent its ability in specialized domains? In this paper, we introduce DomainCodeBench, a multi-domain code generation benchmark designed to systematically evaluate LLMs across 12 software application domains and 15 programming languages. DomainCodeBench contains 2,400 manually verified tasks with ground truth, human-annotated docstrings, and fine-grained dependency information to ensure more coverage of domain-specific challenges. Specifically, we first identify the most popular application domains by topic mining. Then, we curate coding tasks based on commonly used frameworks and platforms in each domain. We obtain several findings through extensive experiments on DomainCodeBench with ten mainstream LLMs. (1) Performance decoupling: experiments reveal that top general-domain models do not consistently excel in specific application domains; (2) Domain-specific weaknesses: LLMs often fail due to domain knowledge gaps and third-party library misusage; (3) Contextual enhancement: we show that augmenting prompts with domain-specific knowledge improves performance by around 38.17%, providing actionable insights for performance optimization. Our replication package, including the benchmark, source code, and experimental results, is available at https://github.com/DeepSoftwareAnalytics/DomainCodeBench.

代码生成多领域评测大模型评估提示工程

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。