NVR通过向量运行预取，显著降低稀疏神经网络的缓存缺失。
NVR: Vector Runahead on NPUs for Sparse Memory Access
- 在NPU上设计轻量级向量运行预取机制，无需编译器支持。
- 相比主流预取方案，缓存缺失减少90%，稀疏任务平均提速4倍。
- 小容量缓存（16KB）配合NVR性能提升是扩大L2缓存的5倍。
深度神经网络正越来越多地利用稀疏性来缓解模型参数规模增长的问题。然而，由于稀疏计算带来的不规则内存访问模式，导致频繁缓存未命中，使得通过稀疏化和剪枝缩短运行时间仍具挑战性。本文提出NPU向量运行预取（NVR），一种专为NPU架构设计的预取机制，以解决稀疏DNN工作负载中的缓存缺失问题。与高开销、可移植性差的内存模式优化不同，NVR将运行预取适配至NPU独特架构。NVR提供通用微架构解决方案，无需编译器或算法支持，作为独立的推测性轻量硬件子线程运行，硬件开销低于5%。在稀疏工作负载上，相比通用处理器的最先进预取方案，缓存缺失平均减少90%，相比无预取的NPU实现4倍平均加速。此外，我们研究了在NPU中集成小缓存（16KB）与NVR的协同优势。评估表明，该小缓存带来的性能提升是同等规模扩展L2缓存的5倍。
原文摘要 · Abstract (English)
Deep Neural Networks are increasingly leveraging sparsity to reduce the scaling up of model parameter size. However, reducing wall-clock time through sparsity and pruning remains challenging due to irregular memory access patterns, leading to frequent cache misses. In this paper, we present NPU Vector Runahead (NVR), a prefetching mechanism tailored for NPUs to address cache miss problems in sparse DNN workloads. Rather than optimising memory patterns with high overhead and poor portability, NVR adapts runahead execution to the unique architecture of NPUs. NVR provides a general micro-architectural solution for sparse DNN workloads without requiring compiler or algorithmic support, operating as a decoupled, speculative, lightweight hardware sub-thread alongside the NPU, with minimal hardware overhead (under 5%). NVR achieves an average 90% reduction in cache misses compared to SOTA prefetching in general-purpose processors, delivering 4x average speedup on sparse workloads versus NPUs without prefetching. Moreover, we investigate the advantages of incorporating a small cache (16KB) into the NPU combined with NVR. Our evaluation shows that expanding this modest cache delivers 5x higher performance benefits than increasing the L2 cache size by the same amount.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。