用CPU实现高效三值大模型推理，不依赖外存查表。
T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganization
- 将SIMD寄存器重用于动态生成查表，避免内存瓶颈。
- 计算延迟降5.6-24.5倍，向量运算吞吐提升1.1-86.2倍。
- 适合资源受限的边缘设备部署，能效比超Jetson AGX Orin。
大语言模型的发展已超越以CPU为主的边缘平台的算力与内存能力，制约其高效可扩展部署。尽管三值量化可大幅节省资源，现有基于CPU的方案仍严重依赖内存中的查表（LUT），限制可扩展性；而FPGA或GPU加速器对边缘场景不实用。本文提出T-SAR，首个通过重新组织SIMD寄存器文件，在寄存器内动态生成查表、仅需少量硬件改动即可实现可扩展三值大模型推理的框架。该方法消除内存瓶颈，最大化数据级并行性，在SIMD单元上仅增加3.2%功耗和1.4%面积开销的前提下，使GEMM延迟降低5.6–24.5倍，GEMV吞吐提升1.1–86.2倍。T-SAR达到NVIDIA Jetson AGX Orin 2.5–4.9倍的能效，为边缘平台提供切实可行的大模型推理方案。
原文摘要 · Abstract (English)
Recent advances in LLMs have outpaced the computational and memory capacities of edge platforms that primarily employ CPUs, thereby challenging efficient and scalable deployment. While ternary quantization enables significant resource savings, existing CPU solutions rely heavily on memory-based lookup tables (LUTs) which limit scalability, and FPGA or GPU accelerators remain impractical for edge use. This paper presents T-SAR, the first framework to achieve scalable ternary LLM inference on CPUs by repurposing the SIMD register file for dynamic, in-register LUT generation with minimal hardware modifications. T-SAR eliminates memory bottlenecks and maximizes data-level parallelism, delivering 5.6-24.5x and 1.1-86.2x improvements in GEMM latency and GEMV throughput, respectively, with only 3.2% power and 1.4% area overheads in SIMD units. T-SAR achieves up to 2.5-4.9x the energy efficiency of an NVIDIA Jetson AGX Orin, establishing a practical approach for efficient LLM inference on edge platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。