通过按需流式加载模型权重，显著降低移动GPU的内存占用和推理延迟。
FlashMem: Supporting Modern DNN Workloads on Mobile with GPU Memory Hierarchy Optimizations
- 静态调度+动态流式加载，利用2.5D纹理内存减少数据转换
- 实测内存减少2.0至8.4倍，速度提升1.7至75.0倍
- 适合在资源受限设备上运行大型或多个模型的场景
现代深度神经网络（DNN）规模与复杂度不断增长，给移动端GPU上的本地推理带来挑战，受限于内存与计算资源。现有加速框架普遍采用权重预加载策略，即执行前将所有模型参数载入内存。我们认为该方法难以应对包含超大规模模型或多模型连续执行的现代工作负载。本文提出FlashMem，一种内存流式框架，可高效执行大规模现代DNN及多模型工作负载，同时最小化内存消耗并降低推理延迟。FlashMem不预先全量加载权重，而是静态确定模型加载调度，并动态按需流式传输，利用2.5D纹理内存减少数据转换，提升执行效率。在11个模型上的实验表明，相比现有框架，FlashMem实现2.0x至8.4x的内存缩减和1.7x至75.0x的速度提升，使资源受限的移动GPU能高效运行大规模模型与多模型任务。
原文摘要 · Abstract (English)
The increasing size and complexity of modern deep neural networks (DNNs) pose significant challenges for on-device inference on mobile GPUs, with limited memory and computational resources. Existing DNN acceleration frameworks primarily deploy a weight preloading strategy, where all model parameters are loaded into memory before execution on mobile GPUs. We posit that this approach is not adequate for modern DNN workloads that comprise very large model(s) and possibly execution of several distinct models in succession. In this work, we introduce FlashMem, a memory streaming framework designed to efficiently execute large-scale modern DNNs and multi-DNN workloads while minimizing memory consumption and reducing inference latency. Instead of fully preloading weights, FlashMem statically determines model loading schedules and dynamically streams them on demand, leveraging 2.5D texture memory to minimize data transformations and improve execution efficiency. Experimental results on 11 models demonstrate that FlashMem achieves 2.0x to 8.4x memory reduction and 1.7x to 75.0x speedup compared to existing frameworks, enabling efficient execution of large-scale models and multi-DNN support on resource-constrained mobile GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。