小模型本地推理，关键词由大模型云端协助，提升性能同时节省资源。
Token Level Routing Inference System for Edge Devices
- 小模型本地运行，仅在关键位置调用云端大模型生成
- 在M1 MacBook上实现60%性能提升，仅0.5%令牌上传云端
- 适合资源受限设备部署，兼顾速度与生成质量
大型语言模型（LLM）推理的计算复杂度严重制约其在边缘设备上的部署效率。相比之下，小型语言模型虽具备更快的解码速度和更低的资源消耗，但常因响应质量下降和幻觉现象频发而受限。为解决这一权衡问题，协同解码——即由大模型在关键令牌生成中提供辅助——成为有前景的解决方案。该范式通过选择性地引入大模型能力，实现高质量推理的同时保持小模型的速度与效率。本文提出一种新型协同解码推理系统，使小模型可在本地完成推理，并仅在必要时调用云端大模型生成关键令牌。实验表明，在M1 MacBook上使用0.5B小模型，于CommonsenseQA数据集上实现了60%的性能提升，且仅有不到7%的令牌生成需上传至云端大模型。
原文摘要 · Abstract (English)
The computational complexity of large language model (LLM) inference significantly constrains their deployment efficiency on edge devices. In contrast, small language models offer faster decoding and lower resource consumption but often suffer from degraded response quality and heightened susceptibility to hallucinations. To address this trade-off, collaborative decoding, in which a large model assists in generating critical tokens, has emerged as a promising solution. This paradigm leverages the strengths of both model types by enabling high-quality inference through selective intervention of the large model, while maintaining the speed and efficiency of the smaller model. In this work, we present a novel collaborative decoding inference system that allows small models to perform on-device inference while selectively consulting a cloud-based large model for critical token generation. Remarkably, the system achieves a 60% performance gain on CommonsenseQA using only a 0.5B model on an M1 MacBook, with under 7% of tokens generation uploaded to the large model in the cloud.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。