通过缓存分片与动态调度，提升多任务深度神经网络在集成NPU上的运行效率
CaMDN: Enhancing Cache Efficiency for Multi-tenant DNNs on Integrated NPUs
- 设计专用缓存区域+智能调度，避免多任务间缓存争用
- 平均减少33.4%内存访问，最高提速2.56倍
- 适合部署多任务深度学习的嵌入式NPU系统
随着深度神经网络应用的快速发展，多个DNN在同一SoC上共存的多租户执行已成为主流趋势。尽管已有诸多方法提升多租户性能，但共享缓存的影响仍未被充分研究。本文提出CaMDN，一种面向集成NPU上多租户DNN的架构-调度协同设计，以提升缓存效率。具体而言，提出轻量级架构，在共享缓存中支持模型专属、NPU控制的区域，消除意外的缓存争用；同时设计缓存调度方法，提升共享缓存利用率。该方法包含适应可变缓存容量的缓存感知映射策略，以及运行时动态调整共存DNN间缓存分配的算法。相比现有工作，CaMDN平均减少33.4%的内存访问，最高实现2.56倍（平均1.88倍）的模型加速。
原文摘要 · Abstract (English)
With the rapid development of DNN applications, multi-tenant execution, where multiple DNNs are co-located on a single SoC, is becoming a prevailing trend. Although many methods are proposed in prior works to improve multi-tenant performance, the impact of shared cache is not well studied. This paper proposes CaMDN, an architecture-scheduling co-design to enhance cache efficiency for multi-tenant DNNs on integrated NPUs. Specifically, a lightweight architecture is proposed to support model-exclusive, NPU-controlled regions inside shared cache to eliminate unexpected cache contention. Moreover, a cache scheduling method is proposed to improve shared cache utilization. In particular, it includes a cache-aware mapping method for adaptability to the varying available cache capacity and a dynamic allocation algorithm to adjust the usage among co-located DNNs at runtime. Compared to prior works, CaMDN reduces the memory access by 33.4% on average and achieves a model speedup of up to 2.56$\times$ (1.88$\times$ on average).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。