用自监督预训练减少代码性能模型的数据需求
Data-efficient Performance Modeling via Pre-training
- 用自编码器在大量随机程序上预训练，学习代码与变换的表示
- 仅需360万数据点就达到22.44%的MAPE，数据量降低5倍
- 适合需要高效构建性能模型的编译器优化研究者
性能模型对自动代码优化至关重要，使编译器能够预测代码变换对性能的影响并指导最优变换搜索。然而，当前基于深度学习的先进性能模型需要大量标注的随机程序数据集，构建过程耗时数月。本文提出一种基于自编码器的自监督预训练方案，通过在大规模随机程序数据上预训练，使模型学会代码与变换的表示，并用于嵌入程序以构建性能模型。该方法在Tiramisu自动调度器中实现，显著降低了对标注数据的需求。例如，原模型需1800万数据点才能达到20.72%的MAPE，而本方法仅用360万数据点即达22.44%的MAPE，数据需求减少5倍。
原文摘要 · Abstract (English)
Performance models are essential for automatic code optimization, enabling compilers to predict the effects of code transformations on performance and guide search for optimal transformations. Building state-of-the-art performance models with deep learning, however, requires vast labeled datasets of random programs -- an expensive and time-consuming process, stretching over months. This paper introduces a self-supervised pre-training scheme with autoencoders to reduce the need for labeled data. By pre-training on a large dataset of random programs, the autoencoder learns representations of code and transformations, which are then used to embed programs for the performance model. Implemented in the Tiramisu autoscheduler, our approach improves model accuracy with less data. For example, to achieve a MAPE of 20.72%, the original model requires 18 million data points, whereas our method achieves a similar MAPE of 22.44% with only 3.6 million data points, reducing data requirements by 5x.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。