TeLLMe在边缘FPGA上实现1.58比特量化大模型，兼顾推理与预填充效率。
TeLLMe: An Energy-Efficient Ternary LLM Accelerator for Prefilling and Decoding on Edge FPGAs
- 采用查表矩阵引擎与在线预计算，降低三值乘法资源开销。
- 支持7W功耗下1024词上下文生成，预填充延迟仅0.55–1.15秒。
- 适合低功耗边缘设备部署生成式AI，尤其关注预填充性能的场景。
将大语言模型（LLMs）部署在边缘平台面临高计算与内存需求挑战。尽管近期低比特量化方法（如BitNet、DeepSeek）可将权重压缩至1.58比特且精度损失极小，但边缘部署仍受限于片上资源、功耗预算，以及常被忽略的预填充阶段延迟。本文提出TeLLMe，首个面向低功耗FPGA（如AMD KV260）的三值LLM加速器，支持1.58比特权重与8比特激活的预填充及自回归解码。贡献包括：(1) 基于查表的矩阵引擎，通过分组激活合并与在线预计算最小化资源占用；(2) 融合的带宽高效注意力模块，采用反向重排策略加速预填充；(3) 紧耦合的归一化与量化-反量化单元，专为超低比特推理优化。在7W功耗下，TeLLMe在1,024词上下文中实现最高9 tokens/s吞吐量，64–128词提示的预填充延迟为0.55–1.15秒，显著提升能效，树立边缘FPGA生成式AI新基准。
原文摘要 · Abstract (English)
Deploying large language models (LLMs) on edge platforms is challenged by their high computational and memory demands. Although recent low-bit quantization methods (e.g., BitNet, DeepSeek) compress weights to as little as 1.58 bits with minimal accuracy loss, edge deployment is still constrained by limited on-chip resources, power budgets, and the often-neglected latency of the prefill phase. We present TeLLMe, the first ternary LLM accelerator for low-power FPGAs (e.g., AMD KV260) that fully supports both prefill and autoregressive decoding using 1.58-bit weights and 8-bit activations. Our contributions include: (1) a table-lookup matrix engine for ternary matmul that merges grouped activations with online precomputation to minimize resource use; (2) a fused, bandwidth-efficient attention module featuring a reversed reordering scheme to accelerate prefill; and (3) a tightly integrated normalization and quantization--dequantization unit optimized for ultra-low-bit inference. Under a 7W power budget, TeLLMe delivers up to 9 tokens/s throughput over 1,024-token contexts and prefill latencies of 0.55--1.15 s for 64--128 token prompts, marking a significant energy-efficiency advance and establishing a new edge FPGA benchmark for generative AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。