为手机端大模型推测解码设计异步混合架构,提升效率与能效。
AHASD: Asynchronous Heterogeneous Architecture for LLM Adaptive Drafting Speculative Decoding on Mobile Devices
- 通过任务级解耦在NPU和存算一体单元并行处理草案生成与验证。
- 动态控制草案生成与预验证时机,降低低置信度草案带来的无效计算。
- 支持亚微秒级任务切换,适用于移动端高能效推理场景。
推测解码通过小模型生成草案,并由大模型批量验证,可提升大语言模型(LLM)的推理效率。然而,在移动单NPU-PIM系统上,传统操作级同步执行存在空闲开销,异步执行则因草案长度波动导致计算浪费。本文提出AHASD,一种面向推测解码的任务级异步移动NPU-PIM异构架构。其通过任务级DLM-TLM解耦,实现PIM端并行草案生成与单NPU端验证;引入基于熵与历史的草案控制及时间感知预验证控制,动态管理自适应草案算法执行与预验证时机,抑制低置信度草案引发的无效计算。同时,将注意力算法单元与门控任务调度单元集成于LPDDR5-PIM中,实现注意力链路定位与亚微秒级任务切换。实验结果表明,针对不同LLM与自适应草案算法,AHASD相比纯GPU基线最高提升4.2×吞吐量与5.6×能效,相较当前最优的GPU+PIM基线提升1.5×吞吐量与1.24×能效,硬件开销低于DRAM面积的3%。
原文摘要 · Abstract (English)
Speculative decoding enhances the inference efficiency of large language models (LLMs) by generating drafts using a small draft language model (DLM) and verifying them in batches with a large target language model (TLM). However, adaptive drafting inference on a mobile single-NPU-PIM system faces idle overhead in traditional operator-level synchronous execution and wasted computation in asynchronous execution due to fluctuations in draft length. This paper introduces AHASD, a task-level asynchronous mobile NPU-PIM heterogeneous architecture for speculative decoding. Notably, AHASD achieves parallel drafting on the PIM and verification on a single NPU through task-level DLM-TLM decoupling and specifically, it incorporates Entropy-History-Aware Drafting Control and Time-Aware Pre-Verification Control to dynamically manage adaptive drafting algorithm execution and pre-verification timing, suppressing invalid drafting based on low-confidence drafts. Additionally, AHASD integrates Attention Algorithm Units and Gated Task Scheduling Units within LPDDR5-PIM to enable attention link localization and sub-microsecond task switching on the PIM side. Experimental results for different LLMs and adaptive drafting algorithms show that AHASD achieves up to 4.2$\times$ in throughput and 5.6$\times$ in energy efficiency improvements over a GPU-only baseline, and 1.5$\times$ in throughput and 1.24$\times$ in energy efficiency gains over the state-of-the-art GPU+PIM baseline, with hardware overhead below 3% of the DRAM area.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。