通过感知模型不确定性,让手机端小模型自主跳过低风险生成,提升推理速度。
Uncertainty-Aware Hybrid Inference with On-Device Small and Remote Large Language Models
- 小模型本地测输出不确定性,高置信度时跳过上传与远程大模型计算
- 实测减少45.93%上行传输和计算量,保持97.54%大模型精度
- 适合移动端低延迟、低功耗语言生成场景
本文研究一种混合语言模型(HLM)架构,将运行在移动设备上的小语言模型(SLM)与无线网络基站(BS)处的大型语言模型(LLM)结合。HLM采用推测性推理机制:由SLM生成的词表分布上传至LLM,LLM决定接受或拒绝,被拒绝的词则由LLM重新采样。该方法虽保证词表分布对齐,但因上行传输和双模型计算开销导致吞吐率低下。为此,本文提出不确定性感知的机遇式混合模型(U-HLM),即SLM本地评估输出不确定性,对高置信度的词跳过上行传输与LLM运算。该策略基于我们发现的SLM不确定性与LLM拒绝概率之间的线性相关性。我们推导出不确定性阈值并分析其预期拒绝风险。仿真表明,U-HLM可减少45.93%的上行传输和LLM计算量,达到大模型97.54%的推理准确率,并实现2.54倍于无跳过的HLM的令牌生成速率。
原文摘要 · Abstract (English)
This paper studies a hybrid language model (HLM) architecture that integrates a small language model (SLM) operating on a mobile device with a large language model (LLM) hosted at the base station (BS) of a wireless network. The HLM token generation process follows the speculative inference principle: the SLM's vocabulary distribution is uploaded to the LLM, which either accepts or rejects it, with rejected tokens being resampled by the LLM. While this approach ensures alignment between the vocabulary distributions of the SLM and LLM, it suffers from low token throughput due to uplink transmission and the computation costs of running both language models. To address this, we propose a novel HLM structure coined Uncertainty-aware opportunistic HLM (U-HLM), wherein the SLM locally measures its output uncertainty and skips both uplink transmissions and LLM operations for tokens that are likely to be accepted. This opportunistic skipping is enabled by our empirical finding of a linear correlation between the SLM's uncertainty and the LLM's rejection probability. We analytically derive the uncertainty threshold and evaluate its expected risk of rejection. Simulations show that U-HLM reduces uplink transmissions and LLM computations by 45.93%, while achieving up to 97.54% of the LLM's inference accuracy and 2.54$\times$ faster token throughput than HLM without skipping.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。