MLPrE是一个可扩展的机器学习数据预处理与分析工具，支持多格式数据流水线处理。
MLPrE -- A tool for preprocessing and exploratory data analysis prior to machine learning model construction
- 基于SparkDataFrames实现可扩展的数据处理，支持多种数据格式
- 内置69个处理阶段，涵盖过滤、统计、特征工程和探索性分析
- 适合需要快速构建数据流水线的研究者或工程师使用
随着深度学习在人工智能中的广泛应用，对数据流入模型前的预处理需求日益增长。原始数据常以多种格式存在，需进行调查与工程化处理，以适配机器学习模型或图数据库。现有流程存在开销大、难扩展等问题，难以集成至如Apache Airflow等大型处理管道中。为此，我们提出机器学习预处理与探索性数据分析工具MLPrE，利用SparkDataFrames确保处理过程的可扩展性，并采用通用的JSON输入格式描述数据框的逐步变换。工具包含输入输出、过滤、基础统计、特征工程及探索性分析等69个处理阶段。通过六个不同数据集验证了关键阶段的有效性。特别展示了对扁平文件中多个字段独立处理并重组的能力，避免额外管道需求；以UniProt术语集数据为例。进一步用葡萄酒质量数据演示聚类阶段。最后，利用磷酸化位点激酶数据，展示如何为图数据库准备数据。总体而言，MLPrE提供了一个通用且可扩展的早期数据处理与分析工具，填补了当前机器学习应用中对此类工具的需求空白，有助于加速和简化大规模工作流中的前期开发。
原文摘要 · Abstract (English)
With the recent growth of Deep Learning for AI, there is a need for tools to meet the demand of data flowing into those models. In some cases, source data may exist in multiple formats, and therefore the source data must be investigated and properly engineered for a Machine Learning model or graph database. Overhead and lack of scalability with existing workflows limit integration within a larger processing pipeline such as Apache Airflow, driving the need for a robust, extensible, and lightweight tool to preprocess arbitrary datasets that scales with data type and size. To address this, we present Machine Learning Preprocessing and Exploratory Data Analysis, MLPrE, in which SparkDataFrames were utilized to hold data during processing and ensure scalability. A generalizable JSON input file format was utilized to describe stepwise changes to that DataFrame. Stages were implemented for input and output, filtering, basic statistics, feature engineering, and exploratory data analysis. A total of 69 stages were implemented into MLPrE, of which we highlight and demonstrate key stages using six diverse datasets. We further highlight MLPrE's ability to independently process multiple fields in flat files and recombine them, otherwise requiring an additional pipeline, using a UniProt glossary term dataset. Building on this advantage, we demonstrated the clustering stage with available wine quality data. Lastly, we demonstrate the preparation of data for a graph database in the final stages of MLPrE using phosphosite kinase data. Overall, our MLPrE tool offers a generalizable and scalable tool for preprocessing and early data analysis, filling a critical need for such a tool given the ever expanding use of machine learning. This tool serves to accelerate and simplify early stage development in larger workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。