让NPU主导大模型推理，减少对CPU/GPU依赖。
ShadowNPU: System and Algorithm Co-design for NPU-Centric On-Device LLM Inference
- 用NPU辅助估算关键令牌，仅稀疏计算部分注意力
- 在极少调用CPU/GPU情况下实现与顶尖框架相当性能
- 适合资源受限的设备端大模型部署
在设备端运行大语言模型（LLMs）是保护用户隐私的关键。我们发现，在现有主流框架中，由于量化敏感性，注意力运算不得不回退到通用的CPU/GPU上执行，导致用户体验下降和系统调度复杂化。为此，本文提出shadowAttn——一种系统与算法协同设计的稀疏注意力模块，通过仅在极少数关键令牌上稀疏计算注意力，极大降低对CPU/GPU的依赖。核心思想是利用NPU进行预计算，隐藏估算重要令牌的开销。同时引入NPU计算图分组、头级流水线及每头细粒度稀疏率等技术，在保持高精度的同时提升效率。shadowAttn在极度有限的CPU/GPU资源下实现最优性能，所需资源远少于当前主流框架即可达到相近表现。
原文摘要 · Abstract (English)
On-device running Large Language Models (LLMs) is nowadays a critical enabler towards preserving user privacy. We observe that the attention operator falls back from the special-purpose NPU to the general-purpose CPU/GPU because of quantization sensitivity in state-of-the-art frameworks. This fallback results in a degraded user experience and increased complexity in system scheduling. To this end, this paper presents shadowAttn, a system-algorithm codesigned sparse attention module with minimal reliance on CPU/GPU by only sparsely calculating the attention on a tiny portion of tokens. The key idea is to hide the overhead of estimating the important tokens with a NPU-based pilot compute. Further, shadowAttn proposes insightful techniques such as NPU compute graph bucketing, head-wise NPU-CPU/GPU pipeline and per-head fine-grained sparsity ratio to achieve high accuracy and efficiency. shadowAttn delivers the best performance with highly limited CPU/GPU resource; it requires much less CPU/GPU resource to deliver on-par performance of SoTA frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。