用纯整数运算加速三值语言模型，让电脑本地跑大模型更快更省内存。
Litespark Inference For CPUs: Ultra-Fast SIMD Framework for Ternary (1.58-bit) Language Models
- 用CPU的整数指令替代浮点乘法，实现三值模型高效推理
- 在苹果芯片上快18.15倍，内存减少6.03倍，首字延迟快7.15倍
- 支持Hugging Face，可一键安装，适合想本地运行大模型的开发者
大型语言模型（LLMs）已重塑人工智能，但其计算需求对多数用户仍显昂贵。标准推理依赖昂贵的数据中心GPU或云API，导致超过十亿台个人电脑无法用于AI任务。三值模型提供了一条可行路径：权重被限制为{-1, 0, +1}，理论上可消除浮点乘法需求。然而现有框架未能利用这一结构，仍将三值模型当作密集浮点网络处理。我们提出定制的SIMD内核，将矩阵乘法替换为加减运算，针对现代CPU的整数点积指令进行优化。所实现的Litespark-Inference可一键安装，与Hugging Face无缝集成，在苹果硅芯片上实现18.15倍吞吐提升、7.15倍首字延迟降低、6.03倍内存减少；在英特尔和AMD处理器上最高可达95.81倍吞吐加速，性能相当或更优。
原文摘要 · Abstract (English)
Large language models (LLMs) have transformed artificial intelligence, but their computational requirements remain prohibitive for most users. Standard inference demands expensive datacenter GPUs or cloud API access, leaving over one billion personal computers underutilized for AI workloads. Ternary models offer a path forward: their weights are constrained to {-1, 0, +1}, theoretically eliminating the need for floating-point multiplication. However, existing frameworks fail to exploit this structure, treating ternary models as dense floating-point networks. We address this gap with custom SIMD kernels that replace matrix multiplication with simple addition and subtraction operations, targeting the integer dot product instructions available on modern CPUs. Our implementation, Litespark-Inference, is pip-installable and integrates directly with Hugging-Face, achieving 18.15x higher throughput, 7.15x faster time-to-first-token and 6.03x memory reduction compared to standard PyTorch inference on Apple Silicon, with comparable or higher throughput speedups up to 95.81x on Intel and AMD processors.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。