让英特尔笔记本的NPU跑起大模型，支持实时聊天生成。
NITRO: LLM Inference on Intel Laptop NPUs
- 基于OpenVINO构建Python框架，改造Transformer结构实现动态生成。
- 在Meteor Lake处理器上实现每秒生成15+个词的推理速度。
- 适合想在本地笔记本上部署大模型的开发者和研究者。
大型语言模型（LLMs）已成为自然语言处理的核心工具，广泛应用于ChatGPT、Gemini等聊天机器人中，是当前研究热点。硬件层面，神经处理单元（NPU）正被专门用于AI应用加速。2023年，英特尔推出代号为Meteor Lake的Core Ultra处理器，集成CPU、GPU与NPU的片上系统。然而，官方通过OpenVINO框架对NPU的支持仅限于静态模型推理，无法直接处理LLM自回归生成所需的动态逐词输出。为此，我们提出NITRO（NPU Inference for Transformers Optimization），一个基于OpenVINO的Python框架，支持在NPU上进行文本与对话生成。本文详细阐述了为适配NPU而对Transformer架构的关键修改、性能基准测试结果及未来优化方向。NITRO代码库见：https://github.com/abdelfattah-lab/nitro。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become essential tools in natural language processing, finding large usage in chatbots such as ChatGPT and Gemini, and are a central area of research. A particular area of interest includes designing hardware specialized for these AI applications, with one such example being the neural processing unit (NPU). In 2023, Intel released the Intel Core Ultra processor with codename Meteor Lake, featuring a CPU, GPU, and NPU system-on-chip. However, official software support for the NPU through Intel's OpenVINO framework is limited to static model inference. The dynamic nature of autoregressive token generation in LLMs is therefore not supported out of the box. To address this shortcoming, we present NITRO (NPU Inference for Transformers Optimization), a Python-based framework built on top of OpenVINO to support text and chat generation on NPUs. In this paper, we discuss in detail the key modifications made to the transformer architecture to enable inference, some performance benchmarks, and future steps towards improving the package. The code repository for NITRO can be found here: https://github.com/abdelfattah-lab/nitro.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。