通过虚拟张量技术，彻底消除深度神经网络编译中的冗余数据搬运。
VTC: DNN Compilation with Virtual Tensors for Data Movement Elimination
- 用虚拟张量追踪计算间数据流动，避免频繁读写全局内存。
- 在NVIDIA GPU上平均提速1.28倍，最高达1.93倍，内存节省17.5%。
- 适合大模型推理优化，尤其擅长处理复杂张量操作组合。
随着计算与内存延迟差距扩大，数据移动优化对深度神经网络（DNN）编译愈发关键。现有优化如布局变换和算子融合仅覆盖部分张量操作，难以捕捉现代DNN工作负载（包括大语言模型）中重要的数据移动机会。本文提出VTC，首个能全面消除非必要数据移动的张量编译框架。VTC引入虚拟张量概念，通过索引映射跟踪计算算子间的数据流动，而非依赖昂贵的全局内存读写，可无缝对接现有计算内核，并支持任意张量操作组合。同时提出一种新型数据移动消除算法，自动识别高效的虚拟张量创建策略。在多种DNN上评估显示，VTC在NVIDIA GPU上相比现有机器学习编译器最高提速1.93倍（平均1.28倍），推理内存消耗最多降低60%（平均17.5%）。
原文摘要 · Abstract (English)
With the widening gap between compute and memory operation latencies, data movement optimizations have become increasingly important for DNN compilation. Current optimizations such as layout transformations and operator fusion only target a subset of tensor operators and consequently miss important opportunities for reducing data movement in contemporary DNN workloads, including large language models. We introduce VTC, a novel tensor compilation framework that for the first time eliminates all unnecessary data movement by targeting the full spectrum of data movement operators. VTC proposes the concept of virtual tensors to track data movement between compute operators via index mappings rather than expensive physical data transfers to and from global memory, which can seamlessly interoperate with existing computation kernels and handle arbitrary tensor operator compositions. We also introduce a novel data movement elimination algorithm to automatically identify a profitable virtual tensor creation strategy. Evaluation on a variety of DNNs shows that VTC can outperform existing ML compilers by up to 1.93x (1.28x on average) on NVIDIA GPUs with up to 60% (17.5% on average) inference memory savings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。