构建首个面向大型C/C++代码库的长上下文生成基准
YABLoCo: Yet Another Benchmark for Long Context Code Generation
- 聚焦百万行级C/C++项目中的函数体生成任务
- 涵盖20万至200万行代码的4个真实开源库
- 提供可扩展评估流水线与生成代码可视化工具
大语言模型在代码生成等编程任务中表现出色，但现有评测基准多基于数千行规模的上下文。本文填补这一空白，提出YABLoCo基准，专为大型代码库的长上下文代码生成设计。该基准包含从4个大型仓库中选取的215个函数，每个仓库规模在20万至200万行代码之间，覆盖函数元信息、不同依赖层级的上下文、文档字符串、函数体及调用图。本研究贡献三方面：一、针对此前未被覆盖的C和C++语言；二、使用真实大规模仓库数据；三、提供可扩展的评估流水线与生成代码可视化分析工具。整体支持对大规模代码库中代码生成能力的系统性评测。
原文摘要 · Abstract (English)
Large Language Models demonstrate the ability to solve various programming tasks, including code generation. Typically, the performance of LLMs is measured on benchmarks with small or medium-sized context windows of thousands of lines of code. At the same time, in real-world software projects, repositories can span up to millions of LoC. This paper closes this gap by contributing to the long context code generation benchmark (YABLoCo). The benchmark featured a test set of 215 functions selected from four large repositories with thousands of functions. The dataset contained metadata of functions, contexts of the functions with different levels of dependencies, docstrings, functions bodies, and call graphs for each repository. This paper presents three key aspects of the contribution. First, the benchmark aims at function body generation in large repositories in C and C++, two languages not covered by previous benchmarks. Second, the benchmark contains large repositories from 200K to 2,000K LoC. Third, we contribute a scalable evaluation pipeline for efficient computing of the target metrics and a tool for visual analysis of generated code. Overall, these three aspects allow for evaluating code generation in large repositories in C and C++.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。