构建18语言代码补全基准，支持细粒度评估与多语言模型训练。
M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation
- 构建覆盖18种语言的代码补全评测集，支持跨语言能力评估。
- 提供桶级与语义级细粒度标注，基于抽象语法树解析生成。
- 配套多语言指令数据集，提升代码大模型跨语言补全能力。
仓库级代码补全在软件工程中备受关注，已有多个基准数据集被提出。然而，现有基准通常仅涵盖少于5种编程语言，难以评估现有代码大语言模型（LLMs）在多种语言间的通用代码智能能力。此外，现有基准多报告各语言的平均得分，忽略了不同补全场景下的细粒度表现。为此，我们提出一个覆盖18种编程语言的海量多语言仓库级代码补全基准M2RC-EVAL，提供两种细粒度标注（桶级与语义级），基于解析的抽象语法树生成。同时，我们还构建了大规模多语言指令语料库M2RC-INSTRUCT数据集，以增强现有代码大模型在仓库级代码补全中的能力。综合实验结果证明了M2RC-EVAL和M2RC-INSTRUCT的有效性。
原文摘要 · Abstract (English)
Repository-level code completion has drawn great attention in software engineering, and several benchmark datasets have been introduced. However, existing repository-level code completion benchmarks usually focus on a limited number of languages (<5), which cannot evaluate the general code intelligence abilities across different languages for existing code Large Language Models (LLMs). Besides, the existing benchmarks usually report overall average scores of different languages, where the fine-grained abilities in different completion scenarios are ignored. Therefore, to facilitate the research of code LLMs in multilingual scenarios, we propose a massively multilingual repository-level code completion benchmark covering 18 programming languages (called M2RC-EVAL), and two types of fine-grained annotations (i.e., bucket-level and semantic-level) on different completion scenarios are provided, where we obtain these annotations based on the parsed abstract syntax tree. Moreover, we also curate a massively multilingual instruction corpora M2RC- INSTRUCT dataset to improve the repository-level code completion abilities of existing code LLMs. Comprehensive experimental results demonstrate the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。