在边缘设备上高效运行大模型，实现55%压缩率与更快推理速度。
On-Device Qwen2.5: Efficient LLM Inference with Model Compression and Hardware Acceleration
- 用激活感知量化+FPGA加速，大幅压缩模型并提升效率。
- 在Kria KV260上实现5.1词元/秒的推理速度，比基线快82%。
- 适合资源受限的边缘部署场景，如移动终端或物联网设备。
基于Transformer的大语言模型虽显著提升了AI能力，但在边缘设备部署时面临计算需求高、内存带宽瓶颈和能耗大的挑战。本文针对Qwen2.5-0.5B模型，在集成ARM Cortex-A53 CPU与可重构FPGA逻辑的Xilinx Kria KV260边缘平台，提出一种高效部署框架。通过激活感知权重量化（AWQ）结合FPGA加速执行管道，实现了更高的模型压缩率与系统吞吐量。此外，设计了混合执行策略，将计算密集型任务智能卸载至FPGA，CPU处理轻量任务，有效平衡负载并最大化性能。实验表明，该框架相较原始模型实现55.08%的压缩率，输出速率达5.1词元/秒，优于基线的2.8词元/秒。
原文摘要 · Abstract (English)
Transformer-based Large Language Models (LLMs) have significantly advanced AI capabilities but pose considerable challenges for deployment on edge devices due to high computational demands, memory bandwidth constraints, and energy consumption. This paper addresses these challenges by presenting an efficient framework for deploying the Qwen2.5-0.5B model on the Xilinx Kria KV260 edge platform, a heterogeneous system integrating an ARM Cortex-A53 CPU with reconfigurable FPGA logic. Leveraging Activation-aware Weight Quantization (AWQ) with FPGA-accelerated execution pipelines, the proposed approach enhances both model compression rate and system throughput. Additionally, we propose a hybrid execution strategy that intelligently offloads compute-intensive operations to the FPGA while utilizing the CPU for lighter tasks, effectively balancing the computational workload and maximizing overall performance. Our framework achieves a model compression rate of 55.08% compared to the original model and produces output at a rate of 5.1 tokens per second, outperforming the baseline performance of 2.8 tokens per second.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。