小模型也能高效处理图文任务,内存占用不到1GB却超越大模型。
SmolVLM: Redefining small and efficient multimodal models
- 设计轻量架构与高效图像分块策略,降低计算开销。
- 最小模型仅256M参数,推理内存<1GB,性能超300倍大的模型。
- 适合移动端和边缘设备部署,尤其视频理解场景。
大型视觉语言模型(VLMs)表现优异但需大量计算资源,难以在移动和边缘设备上部署。小型VLM通常沿用大模型的设计,如过度图像分块,导致显存浪费,实用性受限。我们提出SmolVLM,一系列专为资源高效推理设计的紧凑多模态模型。通过系统性探索架构配置、分块策略与数据筛选,发现关键设计可显著提升图像与视频任务性能,同时保持极低内存占用。最小模型SmolVLM-256M推理时显存不足1GB,尽管开发时间落后18个月,仍超越300倍大的Idefics-80B模型;最大模型2.2B参数,性能媲美耗时两倍显存的顶尖VLM。SmolVLM不仅处理静态图像,还具备强视频理解能力。结果表明,战略性的架构优化、激进但高效的分块方式及精心构建的训练数据,能大幅提升多模态性能,推动小规模、低功耗部署落地。
原文摘要 · Abstract (English)
Large Vision-Language Models (VLMs) deliver exceptional performance but require significant computational resources, limiting their deployment on mobile and edge devices. Smaller VLMs typically mirror design choices of larger models, such as extensive image tokenization, leading to inefficient GPU memory usage and constrained practicality for on-device applications. We introduce SmolVLM, a series of compact multimodal models specifically engineered for resource-efficient inference. We systematically explore architectural configurations, tokenization strategies, and data curation optimized for low computational overhead. Through this, we identify key design choices that yield substantial performance gains on image and video tasks with minimal memory footprints. Our smallest model, SmolVLM-256M, uses less than 1GB GPU memory during inference and outperforms the 300-times larger Idefics-80B model, despite an 18-month development gap. Our largest model, at 2.2B parameters, rivals state-of-the-art VLMs consuming twice the GPU memory. SmolVLM models extend beyond static images, demonstrating robust video comprehension capabilities. Our results emphasize that strategic architectural optimizations, aggressive yet efficient tokenization, and carefully curated training data significantly enhance multimodal performance, facilitating practical, energy-efficient deployments at significantly smaller scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。