通过智能压缩与择机传输，显著降低语言模型通信开销。
Communication-Efficient Hybrid Language Model via Uncertainty-Aware Opportunistic and Compressed Transmission
- 仅在小模型输出不确定时才上传压缩词汇分布
- 减少74.8%传输量，实现206倍吞吐提升
- 适合资源受限的边缘设备部署
为支持基于分散异构计算资源的语言应用，混合语言模型（HLM）采用本地小型语言模型（SLM）生成候选词元，由远程大型语言模型（LLM）验证并修正。但原始HLM存在巨大通信开销，因LLM需接收每个词元的完整词汇分布。此外，当词元被高度接受时，通信与计算资源仍被浪费。为此，我们提出通信高效且不确定性感知的混合语言模型（CU-HLM）。在CU-HLM中，只有当SLM输出不确定性较高时，才上传截断的词汇分布。我们通过发现SLM不确定性与LLM拒绝概率间的强相关性，验证了该择机传输的可行性。进一步地，我们理论推导出最优不确定性阈值和最优词汇截断策略。仿真结果表明，相较于标准HLM，CU-HLM在跳过74.8%传输、实现97.4%词汇压缩的同时，保持97.4%准确率，并将词元吞吐量提升高达206倍。
原文摘要 · Abstract (English)
To support emerging language-based applications using dispersed and heterogeneous computing resources, the hybrid language model (HLM) offers a promising architecture, where an on-device small language model (SLM) generates draft tokens that are validated and corrected by a remote large language model (LLM). However, the original HLM suffers from substantial communication overhead, as the LLM requires the SLM to upload the full vocabulary distribution for each token. Moreover, both communication and computation resources are wasted when the LLM validates tokens that are highly likely to be accepted. To overcome these limitations, we propose communication-efficient and uncertainty-aware HLM (CU-HLM). In CU-HLM, the SLM transmits truncated vocabulary distributions only when its output uncertainty is high. We validate the feasibility of this opportunistic transmission by discovering a strong correlation between SLM's uncertainty and LLM's rejection probability. Furthermore, we theoretically derive optimal uncertainty thresholds and optimal vocabulary truncation strategies. Simulation results show that, compared to standard HLM, CU-HLM achieves up to 206$\times$ higher token throughput by skipping 74.8% transmissions with 97.4% vocabulary compression, while maintaining 97.4% accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。