让手机和云端协作推理视觉语言模型,速度提升2.21倍且通信减少96%。
CoVSpec: Efficient Device-Edge Co-Inference for Vision-Language Models via Speculative Decoding

- 手机端用剪枝法减少视觉令牌,云端动态调整验证频率和生成长度。
- 在多个基准上实现2.21倍吞吐量提升,通信开销降低超96%。
- 适合移动端部署大模型的开发者,尤其关注效率与带宽的场景。
视觉语言模型(VLMs)在多模态感知与推理方面表现出强大能力,但将其部署于移动设备仍面临巨大计算与内存挑战。一种实用方案是设备-边缘协同推理,即在手机端运行轻量级草稿VLM,通过推测解码与边缘服务器上的大型目标VLM协同工作。然而,直接将推测解码应用于VLM会因过多视觉令牌计算和高通信开销导致严重低效。为此,我们提出CoVSpec,一种高效的协同推测解码框架。首先,设计无需训练的视觉令牌压缩方法,在手机端联合考虑查询相关性、令牌活跃度与低秩依赖关系以剔除冗余视觉令牌;其次,提出自适应草稿策略,动态调节验证频率与草稿长度;此外,引入并行分支机制,分离验证与修正过程,提升草稿端利用率并减少修正阶段传输开销。在多个基准测试中,CoVSpec相较仅使用目标模型推理提升最高2.21倍吞吐量,通信开销相比基线降低超过96%,且不损失任务准确率。
原文摘要 · Abstract (English)
Vision-language models (VLMs) have demonstrated strong capabilities in multimodal perception and reasoning. However, deploying large VLMs on mobile devices remains challenging due to their substantial computational and memory demands. A practical alternative is device-edge co-inference, where a lightweight draft VLM on the mobile device collaborates with a larger target VLM on the edge server via speculative decoding. Nevertheless, directly extending speculative decoding to VLMs suffers from severe inefficiency due to excessive visual-token computation and high communication overhead. To address these challenges, we propose CoVSpec, an efficient collaborative speculative decoding framework for VLM inference. Specifically, we first develop a training-free visual token reduction framework that prunes redundant visual tokens on the mobile device by jointly considering query relevance, token activity, and low-rank dependency. Moreover, we design an adaptive drafting strategy that dynamically adjusts both the verification frequency and the draft length. In addition, we introduce a parallel branching mechanism with decoupled verification-correction to improve draft-side utilization during target-side verification and reduce correction-related transmission overhead. Experiments on multiple benchmarks show that CoVSpec achieves up to 2.21x higher throughput than target-only inference and reduces communication overhead by more than 96% compared with baselines, without compromising task accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。