基于Stack Overflow构建代码生成数据集，含真实开发场景与测试用例。
CodeInsight: A Curated Dataset of Practical Coding Solutions from Stack Overflow
- 精选Python常见任务的代码示例，每例配明确意图与平均3个单元测试
- 涵盖3409个专家撰写样本，覆盖Pandas、Numpy等70+标准库
- 专为模型微调与细粒度评估设计，适合研究代码生成与测试能力
我们引入一个面向代码生成的新颖数据集，旨在帮助开发者完成常见任务。该数据集包含经过清晰意图标注的代码片段，平均每例配有三个相关单元测试，涵盖 exttt{Pandas}、 exttt{Numpy} 和 exttt{Regex} 等库，以及从 Stack Overflow 提取的超过70个标准 Python 库。数据由Python专家精心制作，共3,409个示例，适用于模型微调和独立评估。为实现更细粒度的测试分析，我们对示例进行分类，以更好理解模型在特定编码任务中的优劣。数据已通过去污染处理，经三款主流模型（Mistral 7B、CodeLLaMa 13B、Starcoder 15B）验证其有效性。我们还对GPT-4在部分数据上的表现进行了数据污染测试。基准数据可在 exttt{https://github.com/NathanaelBeau/CodeInsight} 获取。
原文摘要 · Abstract (English)
We introduce a novel dataset tailored for code generation, aimed at aiding developers in common tasks. Our dataset provides examples that include a clarified intent, code snippets associated, and an average of three related unit tests. It encompasses a range of libraries such as \texttt{Pandas}, \texttt{Numpy}, and \texttt{Regex}, along with more than 70 standard libraries in Python code derived from Stack Overflow. Comprising 3,409 crafted examples by Python experts, our dataset is designed for both model finetuning and standalone evaluation. To complete unit tests evaluation, we categorize examples in order to get more fine grained analysis, enhancing the understanding of models' strengths and weaknesses in specific coding tasks. The examples have been refined to reduce data contamination, a process confirmed by the performance of three leading models: Mistral 7B, CodeLLaMa 13B, and Starcoder 15B. We further investigate data-contamination testing GPT-4 performance on a part of our dataset. The benchmark can be accessed at \url{https://github.com/NathanaelBeau/CodeInsight}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。