新基准KOCO-BENCH测试大模型在真实开发中获取应用领域知识的能力
KOCO-BENCH: Can Large Language Models Leverage Domain Knowledge in Software Development?
- 构建6个新兴领域、11个框架、25个项目的知识库与多粒度任务
- 顶尖模型仅34.2%准确率，凸显领域专业化方法仍严重不足
- 适合研究模型如何学习和使用领域知识的研究者使用
大型语言模型（LLMs）在通用编程上表现优异，但在特定领域软件开发中仍面临挑战，亟需领域专业化方法来学习并利用领域知识。然而，现有领域代码基准无法评估领域专业化方法的有效性，因其关注的是模型已掌握的知识而非如何获取和应用新知识，且缺乏用于开发领域专业化方法的明确知识语料库。为此，我们提出KOCO-BENCH，一个专为评估真实软件开发中领域专业化方法而设计的新基准。该基准包含6个新兴领域、11个软件框架和25个实际项目，配有精选知识语料库，并设置多粒度评估任务，涵盖从函数级到项目级的领域代码生成（配有严格测试套件）以及基于多项选择题的领域知识理解。与以往仅提供测试集的基准不同，KOCO-BENCH要求从知识语料库中获取并应用多样化的领域知识（如API、规则、约束等）以完成任务。我们的评估表明，即使使用SFT、RAG、kNN-LM等先进领域专业化方法，性能提升依然有限。表现最佳的编码代理Claude Code仅达34.2%准确率，凸显亟需更有效的领域专业化方法。我们已将KOCO-BENCH、评估代码和基线发布至https://github.com/jiangxxxue/KOCO-bench，以推动后续研究。
原文摘要 · Abstract (English)
Large language models (LLMs) excel at general programming but struggle with domain-specific software development, necessitating domain specialization methods for LLMs to learn and utilize domain knowledge and data. However, existing domain-specific code benchmarks cannot evaluate the effectiveness of domain specialization methods, which focus on assessing what knowledge LLMs possess rather than how they acquire and apply new knowledge, lacking explicit knowledge corpora for developing domain specialization methods. To this end, we present KOCO-BENCH, a novel benchmark designed for evaluating domain specialization methods in real-world software development. KOCO-BENCH contains 6 emerging domains with 11 software frameworks and 25 projects, featuring curated knowledge corpora alongside multi-granularity evaluation tasks including domain code generation (from function-level to project-level with rigorous test suites) and domain knowledge understanding (via multiple-choice Q&A). Unlike previous benchmarks that only provide test sets for direct evaluation, KOCO-BENCH requires acquiring and applying diverse domain knowledge (APIs, rules, constraints, etc.) from knowledge corpora to solve evaluation tasks. Our evaluations reveal that KOCO-BENCH poses significant challenges to state-of-the-art LLMs. Even with domain specialization methods (e.g., SFT, RAG, kNN-LM) applied, improvements remain marginal. Best-performing coding agent, Claude Code, achieves only 34.2%, highlighting the urgent need for more effective domain specialization methods. We release KOCO-BENCH, evaluation code, and baselines to advance further research at https://github.com/jiangxxxue/KOCO-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。