ELiTeFormer通过硬件协同设计，实现超低精度与线性注意力融合，显著提升FPGA部署效率。
ELiTeFormer: An Efficient Transformer for FPGAs
- 采用三值线性投影与混合线性注意力，用位掩码替代乘法，减少硬件资源占用。
- 模型权重压缩10倍，键值缓存压缩12.8倍，MMLU准确率达31.9%，接近BitNet性能。
- 专为FPGA优化，相比GPU在长文本场景下延迟降低3.9倍，能效提升3.2倍。
Transformer模块在大语言模型中广泛应用，但其高计算与内存需求带来部署挑战。现有工作多单独优化注意力或前馈网络，少有硬件架构同时协同优化两者。本文提出ELiTeFormer（高效线性三值Transformer），首个将混合线性注意力与超低精度（三值）线性投影统一设计的Transformer架构，专为现场可编程门阵列（FPGA）部署而生。相比LLaMA 3，ELiTeFormer实现10倍模型权重压缩和12.8倍键值（KV）缓存压缩，同时保持竞争力表现（MMLU基准31.9%准确率，较BitNet b1.58低3.0%）。核心创新在于新型处理单元（PE）微架构，通过位掩码操作完全消除三值线性投影中的乘法运算，大幅降低资源开销，避免使用专用数字信号处理（DSP）块。我们基于高层次综合（HLS）流程，在Xilinx VCK5000 Versal平台上完成仿真、综合与部署。模块级仿真显示，前馈网络加速9.6倍，注意力加速4.4倍。端到端部署在长上下文场景下，相较NVIDIA A100 GPU，延迟降低3.9倍，能效提升3.2倍。这是首个结合线性注意力与三值量化在FPGA上的实现，验证了算法-架构协同设计在下一代大模型加速中的可行性。
原文摘要 · Abstract (English)
Transformer blocks are prevalent in large language model (LLM) but present deployment challenges due to their challenging computational and memory demands. While prior work has typically optimized attention mechanisms or feed-forward networks (FFNs) separately, few hardware (HW) architecture have jointly addressed both components with co-designed hardware acceleration. We present ELiTeFormer (Efficient Linear Ternary Transformer), the first Transformer model architecture that unifies hybrid linear attention with ultra-low-precision (ternary) linear projections, specifically co-designed for field-programmable gate array (FPGA) deployment. ELiTeFormer achieves 10x model weight compression and 12.8x key-value (KV) cache compression compared to LLaMA 3, while maintaining competitive accuracy (31.9% on the MMLU benchmark, within 3.0% of BitNet b1.58). Our key architectural contribution is a novel processing element (PE) micro-architecture that eliminates all multiplications in ternary linear projections through bitmasking operations, significantly reducing resource utilization by completely avoiding dedicated digital signal processing (DSP) blocks. We simulate, synthesize, and deploy ELiTeFormer targeting a Xilinx VCK5000 Versal board using high-level synthesis (HLS) flows. Block-level simulations show 9.6x speedup for FFN operations and 4.4x speedup for attention compared to standard implementations. End-to-end deployment achieves up to 3.9x lower latency and 3.2x better energy efficiency than LLaMA 3 on an NVIDIA A100 graphics processing unit (GPU) at long context lengths. This represents the first FPGA realization combining linear attention with ternary quantization, demonstrating the viability of algorithm-architecture co-design for next-generation LLM acceleration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。