用AI生成高性价比的CUDA跨平台代码对,提升LLM在高性能计算中的移植能力。
HPCTransCompile: An AI Compiler Generated Dataset for High-Performance CUDA Transpilation and LLM Preliminary Exploration
- 用AI编译器自动构建CUDA与目标平台代码对,支持高性能移植。
- 在CPU上实现43.8%的平均性能提升,验证了方法有效性。
- 开源数据集与评估基准,适合研究跨平台编译和LLM代码生成者使用。
深度学习模型参数与计算需求的指数级增长推动了对高性能并行计算的迫切需求。NVIDIA GPU及其基于CUDA的软件生态为并行计算提供了强大支持,显著缓解了计算瓶颈。由于用户编程习惯和GPU的高性能,CUDA生态在并行软件领域占据主导地位,要求其他硬件平台具备对CUDA软件的性能可移植性。然而,由于并行编程范式与硬件架构差异,将CUDA代码迁移到其他平台面临巨大挑战。现有方法依赖语言扩展、领域特定语言(DSL)或编译器,但在工作负载覆盖范围和泛化能力上存在局限,且开发成本高。近期大模型(LLMs)在代码相关任务中展现出巨大潜力,但其在高性能CUDA代码移植中的表现仍不理想。为此,我们提出一种新框架,利用AI编译器与自动优化技术生成高质量的CUDA与目标平台代码对。通过图结构数据增强方法进一步优化,并引入HPCTransEval基准用于评估LLM在CUDA移植任务中的表现。以CUDA到CPU移植为例,实验表明CPU算子平均加速比提升43.8%,证明了大模型在解决CUDA生态兼容性问题上的潜力。代码已开源:https://github.com/PJLAB-CHIP/HPCTransCompile。
原文摘要 · Abstract (English)
The rapid growth of deep learning has driven exponential increases in model parameters and computational demands. NVIDIA GPUs and their CUDA-based software ecosystem provide robust support for parallel computing, significantly alleviating computational bottlenecks. Meanwhile, due to the cultivation of user programming habits and the high performance of GPUs, the CUDA ecosystem has established a dominant position in the field of parallel software. This dominance requires other hardware platforms to support CUDA-based software with performance portability. However, translating CUDA code to other platforms poses significant challenges due to differences in parallel programming paradigms and hardware architectures. Existing approaches rely on language extensions, domain-specific languages (DSLs), or compilers but face limitations in workload coverage and generalizability. Moreover, these methods often incur substantial development costs. Recently, LLMs have demonstrated extraordinary potential in various vertical domains, especially in code-related tasks. However, the performance of existing LLMs in CUDA transpilation, particularly for high-performance code, remains suboptimal. To address these challenges, we propose a novel framework for generating high-performance CUDA and corresponding platform code pairs, leveraging AI compiler and automatic optimization technology. We further enhance the framework with a graph-based data augmentation method and introduce HPCTransEval, a benchmark for evaluating LLM performance on CUDA transpilation. We conduct experiments using CUDA-to-CPU transpilation as a case study on leading LLMs. The speedup ratio of the CPU operators has an average improvemnet of 43.8\%, highlighting the potential of LLMs to address compatibility challenges within the CUDA ecosystem. Our code is available at https://github.com/PJLAB-CHIP/HPCTransCompile.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。