对比三大数据处理库在深度学习全流程中的能耗表现
Energy Consumption of Dataframe Libraries for End-to-End Deep Learning Pipelines:A Comparative Analysis
- 在完整训练推理流程中测试Pandas、Polars、Dask的性能
- 发现Polars在多数场景下能耗最低，运行速度最快
- 适合关注能效的深度学习系统开发者参考
本文针对Pandas、Polars和Dask三大主流Python数据处理库，在完整的深度学习训练与推理流水线中的表现进行了详尽的对比分析。研究填补了现有文献的空白，重点考察这些库在数据加载、预处理和批处理等关键阶段与大规模GPU任务的交互情况。作者测量了运行时间、内存占用、磁盘使用量以及CPU和GPU的能耗等关键性能指标，覆盖多种机器学习模型与数据集。
原文摘要 · Abstract (English)
This paper presents a detailed comparative analysis of the performance of three major Python data manipulation libraries - Pandas, Polars, and Dask - specifically when embedded within complete deep learning (DL) training and inference pipelines. The research bridges a gap in existing literature by studying how these libraries interact with substantial GPU workloads during critical phases like data loading, preprocessing, and batch feeding. The authors measured key performance indicators including runtime, memory usage, disk usage, and energy consumption (both CPU and GPU) across various machine learning models and datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。