构建首个大规模真实Pandas工作流数据集，揭示数据分析实践中的使用模式。
PandasCorpus: A Resource of Real-World Pandas Workflows and Usage Patterns
- 从10万仓库中提取13.9万份Jupyter笔记本，捕获超400万次Pandas调用
- 发现常用操作序列与代码可执行性随时间演变的规律
- 适合研究数据科学工具使用、代码生成或编程教育者参考
Pandas已成为数据处理与机器学习的主流库，广泛用于数据加载、转换与分析。然而，其在真实项目中的使用方式及典型工作流组成仍缺乏系统研究。为此，我们构建了PandasCorpus，一个从GitHub仓库中提取的真实世界Pandas工作流数据集。该数据集包含约10万个仓库中的13.9万份Jupyter笔记本，涵盖超过400万次Pandas API调用，涉及136种不同操作。通过结构化特征与Pandas特异性指标分析2015至2025年间笔记本演化，研究了代码可执行性、笔记本规模及常见操作序列。结果为理解实际数据分析流程、改进库感知代码生成提供了实证依据。数据集与提取工具已通过GitHub和Zenodo公开。
原文摘要 · Abstract (English)
Pandas has emerged as the de facto library for data processing and machine learning, widely used for tasks, such as data loading, transformation, and analysis. Despite its ubiquity, there has been limited systematic investigation into how Pandas is used in real-world projects and how typical workflows are composed in practice. To address this gap, we introduce PandasCorpus, a dataset curated from GitHub repositories that captures real-world Pandas workflows at scale. In this work, a workflow refers to Pandas-based code contained in Jupyter notebooks, a prevalent medium for writing, executing, and sharing data analysis code. The dataset comprises 139k notebooks from approximately 100k repositories and captures more than 4M Pandas API calls spanning 136 distinct operations. Beyond dataset construction, we characterize workflows using structural and Pandas-specific features and analyze notebook evolution between 2015 and 2025. Our study examines code executability, notebook size, and recurring sequences of Pandas operations, providing empirical insights into how Pandas is used in practice. The resulting corpus offers a reusable resource for studying data analysis workflows, Pandas usage patterns, and library-aware code composition. Both the dataset and the extraction pipeline are publicly available via GitHub and Zenodo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。