提出向量查找表，让边缘设备上低比特大模型推理快4.2倍。
Vec-LUT: Vector Table Lookup for Parallel Ultra-Low-Bit LLM Inference on Edge Devices
- 用统一向量表替代逐个查表，一次查多个并行token
- 在5个边缘设备上实测速度提升最高达4.2倍
- 适合做低比特大模型边缘部署的开发者
大语言模型正越来越多地部署在边缘设备上。为满足严苛的资源限制，实际部署已将模型量化从8比特推进到4比特、2比特，甚至1.58比特。结合查表（LUT）推理方式，CPU运行这些超低比特大模型的速度甚至超过NPU，为无处不在的设备端智能打开新机遇。然而本文发现，现有基于查表的并行推理会浪费内存带宽，这是因为在预填充、多标记生成等场景下，传统的标量查表模式对每个标记重复进行非连续内存访问。为此，我们提出向量查表（Vec-LUT），构建跨并行标记的统一查表结构，实现每索引一次1→N的批量查找。为高效实现，进一步引入（1）向量查表中心张量布局，和（2）缓存感知流式查表技术。在3个大模型、5个边缘设备上的评估显示，Vec-LUT相比当前最佳基线性能最高提升4.2倍。代码已集成至llama.cpp，开源地址：https://github.com/OpenBitSys/vlut.cpp。
原文摘要 · Abstract (English)
Large language models (LLMs) are increasingly deployed on edge devices. To meet strict resource constraints, real-world deployment has pushed LLM quantization from 8-bit to 4-bit, 2-bit, and now 1.58-bit. Combined with lookup table (LUT)-based inference, CPUs run these ultra-low-bit LLMs even faster than NPUs, opening new opportunities for ubiquitous on-device intelligence. However, this paper identifies that LUT-based inference underutilizes memory bandwidth during parallel inference, which is required for prefilling, test-time scaling, and other multi-token scenarios. The root cause is the scalar LUT paradigm, which performs repetitive and non-contiguous memory accesses for each token. To solve the issue, we propose vector LUT, a new lookup paradigm that constructs a unified LUT across parallel tokens, and performs a single $1 \rightarrow N$ lookup per index. To realize it efficiently, we further introduce (1) Vector LUT-Centric Tensor Layout, and (2) Cache-Aware Streamed Lookup techniques. Evaluations on 5 edge devices across 3 LLMs show that Vec-LUT outperforms state-of-the-art baselines by up to $4.2\times$. Our implementation is integrated into llama.cpp. The code is available at https://github.com/OpenBitSys/vlut.cpp.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。