构建首个面向代码库级问答的大规模评测基准，评估大模型在软件工程场景下的表现。
CodeRepoQA: A Large-scale Benchmark for Software Engineering Question Answering
- 从30个知名GitHub仓库爬取数据，构建多语言、多场景的对话式问答集
- 包含58万+条记录，平均每条6.62轮对话，覆盖广泛软件工程任务
- 揭示大模型在代码理解上的不足，提示中等上下文长度更利于性能
本文提出CodeRepoQA，一个专为评估软件工程领域代码库级问答能力而设计的大规模基准。该数据集涵盖五种编程语言，覆盖多样化的实际开发场景。数据源自GitHub上30个知名开源仓库，经严格筛选后构建。整体为多轮对话问答形式，共包含585,687条样本，平均每条对话达6.62轮。我们对十款主流大语言模型进行了评估并深入分析，发现当前大模型在软件工程问答任务中仍存在明显局限，且中等长度上下文更有利于模型表现。该基准已公开于https://github.com/kinesiatricssxilm14/CodeRepoQA。
原文摘要 · Abstract (English)
In this work, we introduce CodeRepoQA, a large-scale benchmark specifically designed for evaluating repository-level question-answering capabilities in the field of software engineering. CodeRepoQA encompasses five programming languages and covers a wide range of scenarios, enabling comprehensive evaluation of language models. To construct this dataset, we crawl data from 30 well-known repositories in GitHub, the largest platform for hosting and collaborating on code, and carefully filter raw data. In total, CodeRepoQA is a multi-turn question-answering benchmark with 585,687 entries, covering a diverse array of software engineering scenarios, with an average of 6.62 dialogue turns per entry. We evaluate ten popular large language models on our dataset and provide in-depth analysis. We find that LLMs still have limitations in question-answering capabilities in the field of software engineering, and medium-length contexts are more conducive to LLMs' performance. The entire benchmark is publicly available at https://github.com/kinesiatricssxilm14/CodeRepoQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。