通过筛选重要与不确定的词元，降低边缘设备大模型推理能耗。
Energy-Efficient Wireless LLM Inference via Uncertainty and Importance-Aware Speculative Decoding
- 根据语义不确定性与注意力重要性筛选词元，只上传关键信息。
- 相比标准混合模型，能耗降低40.7%，吞吐量达0.37词元/秒。
- 适合带宽受限的边缘设备部署，兼顾精度与能效。
为应对资源受限环境下对本地大模型推理日益增长的需求，混合语言模型（HLM）应运而生，结合轻量级本地模型与强大的云端大模型。现有研究多关注准确率与延迟，却常忽视通信与能耗效率。本文提出一种面向能量高效的词元级过滤机制，基于认知不确定性与注意力重要性，智能上传高信息量词元，减少大模型调用与通信开销。在TinyLlama-1.1B和LLaMA-2-7B上的实验表明，该方法可实现高达87.5%的BERT Score，词元吞吐量达0.37 tokens/sec，相比标准HLM节能40.7%。相较于先前的U-HLM基线，本方法将BERTScore从85.8%提升至87.0%，能耗节省从31.6%增至43.6%，吞吐量由0.36提升至0.40。该方案实现了带宽受限边缘环境下的高效、低耗且准确的大模型部署。
原文摘要 · Abstract (English)
To address the growing demand for on-device LLM inference in resource-constrained environments, hybrid language models (HLM) have emerged, combining lightweight local models with powerful cloud-based LLMs. Recent studies on HLM have primarily focused on improving accuracy and latency, while often overlooking communication and energy efficiency. We propose a token-level filtering mechanism for an energy-efficient importance- and uncertainty-aware HLM inference that leverages both epistemic uncertainty and attention-based importance. Our method opportunistically uploads only informative tokens, reducing LLM usage and communication costs. Experiments with TinyLlama-1.1B and LLaMA-2-7B demonstrate that our method achieves up to 87.5% BERT Score and token throughput of 0.37 tokens/sec while saving the energy consumption by 40.7% compared to standard HLM. Furthermore, compared to our previous U-HLM baseline, our method improves BERTScore from 85.8% to 87.0%, energy savings from 31.6% to 43.6%, and throughput from 0.36 to 0.40. This approach enables an energy-efficient and accurate deployment of LLMs in bandwidth-constrained edge environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。