构建32万+真实Python类数据集，助力大模型代码生成能力评估
OpenClassGen: A Large-Scale Corpus of Real-World Python Classes for LLM Research
- 从2970个开源项目提取32万+真实类，含签名与文档注释
- 评估显示模型语义相似度高（CodeBERTScore-F3: 0.89），但功能正确率仅33%
- 支持模型微调、错误分析等研究，适合代码生成与智能编程研究者
现有类级别代码生成数据集或为合成数据（ClassEval：100类），或规模不足（RealClassEval：400类），难以满足现代训练需求。本文提出OpenClassGen，一个从2,970个工程化开源项目中提取的324,843个真实Python类大规模语料库。每个条目包含人工编写的类及其对应的骨架，包括类与方法签名及关联文档注释，并附带27项静态代码度量，涵盖复杂度、耦合度、内聚度和继承特性。不同于需仓库级上下文解析的先前基准，OpenClassGen提供自包含类骨架，可作为完整生成规范。我们通过在300个可执行类子集上评估三个LLM（GPT-o4-mini、Claude-4-Sonnet、Qwen-3-Coder）验证其价值，该子集配备测试套件，实现58%分支覆盖率。结果显示，模型语义相似度较高（CodeBERTScore-F3: 0.89），但功能正确率仅为0.33，且各模型间差异显著。这一差异连同类特征多样性，证明OpenClassGen能有效区分大模型能力。该数据集适用于微调、检索增强生成、难度建模与失败模式分析。完整数据集及清洗脚本已公开于https://zenodo.org/records/18409150。
原文摘要 · Abstract (English)
Existing class-level code generation datasets are either synthetic (ClassEval: 100 classes) or insufficient in scale for modern training needs (RealClassEval: 400 classes), hindering robust evaluation and empirical analysis. We present OpenClassGen, a large-scale corpus of 324,843 Python classes extracted from 2,970 engineered open-source projects. Each entry pairs a human-written class with its corresponding skeleton, which comprises class and method signatures with associated docstrings, and is enriched with 27 static code metrics covering complexity, coupling, cohesion, and inheritance properties. Unlike prior benchmarks that require repository-level context resolution, OpenClassGen provides self-contained class skeletons that serve as complete generation specifications. We demonstrate the corpus's utility by evaluating three LLMs (GPT-o4-mini, Claude-4-Sonnet, Qwen-3-Coder) on a curated, executable subset of 300 classes, enriched with test suites achieving 58% branch coverage. Results show strong semantic similarity (CodeBERTScore-F3: 0.89) but moderate functional correctness (pass rate: 0.33), with substantial variance across models. This variance, along with diverse class characteristics, confirms that OpenClassGen enables meaningful differentiation of LLM capabilities. The dataset supports diverse use cases, including fine-tuning, retrieval-augmented generation, difficulty modelling, and failure mode analysis. The complete dataset and curation scripts are publicly available at https://zenodo.org/records/18409150.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。