让手机能跑的多模态大模型,速度快功耗低。
HyperVL: An Efficient and Dynamic Multimodal Large Language Model for Edge Devices
- 用图像分块+自适应分辨率压缩,降低内存占用
- 多尺度视觉编码器动态切换,同等大小下性能领先
- 适合移动端部署,实测延迟和功耗大幅下降
当前多模态大模型虽具备强大感知与推理能力,但计算与内存开销大,难以直接部署于设备端。尽管小参数模型逐步具备强泛化能力,标准视觉变压器(ViT)编码器在处理高分辨率输入时仍存在严重延迟与内存消耗问题。为此,我们提出HyperVL,一种专为设备端推理设计的高效多模态大模型。HyperVL采用图像分块策略控制峰值内存,并引入两项新技术:(1) 视觉分辨率压缩器(VRC),可自适应预测最优编码分辨率以消除冗余计算;(2) 双一致性学习(DCL),在统一框架内对齐多尺度ViT编码器,实现共享语言模型下的视觉分支动态切换。大量实验表明,HyperVL在多个基准上达到同类规模模型的最先进性能。此外,在真实移动设备上显著降低延迟与功耗,验证了其在设备端多模态推理中的实用性。
原文摘要 · Abstract (English)
Current multimodal large lanauge models possess strong perceptual and reasoning capabilities, however high computational and memory requirements make them difficult to deploy directly on on-device environments. While small-parameter models are progressively endowed with strong general capabilities, standard Vision Transformer (ViT) encoders remain a critical bottleneck, suffering from excessive latency and memory consumption when processing high-resolution inputs.To address these challenges, we introduce HyperVL, an efficient multimodal large language model tailored for on-device inference. HyperVL adopts an image-tiling strategy to cap peak memory usage and incorporates two novel techniques: (1) a Visual Resolution Compressor (VRC) that adaptively predicts optimal encoding resolutions to eliminate redundant computation, and (2) Dual Consistency Learning (DCL), which aligns multi-scale ViT encoders within a unified framework, enabling dynamic switching between visual branches under a shared LLM. Extensive experiments demonstrate that HyperVL achieves state-of-the-art performance among models of comparable size across multiple benchmarks. Furthermore, it significantly significantly reduces latency and power consumption on real mobile devices, demonstrating its practicality for on-device multimodal inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。