压缩视觉令牌，968万参数实现手机端高效图文理解
OmniVLM: A Token-Compressed, Sub-Billion-Parameter Vision-Language Model for Efficient On-Device Inference
- 用令牌压缩技术将视觉序列从729降至81，降低计算开销
- 在ScienceQA等数据集上超越nanoLLAVA，968M参数下表现更优
- 适合边缘设备部署，手机推理速度提升9.1倍
我们提出OmniVLM，一个低于十亿参数的视觉语言模型，专为高效设备端推理设计。OmniVLM引入令牌压缩机制，将视觉令牌序列长度从729缩减至81，显著降低计算开销，同时保持视觉语义一致性。通过多阶段训练流程——预训练、监督微调及最小编辑的直接偏好优化（DPO），OmniVLM性能媲美更大模型。在ScienceQA、POPE和MMMU等多个基准测试中，其在968M参数规模下优于现有基线如nanoLLAVA。实测显示，在同一笔记本电脑上，OmniVLM的首令牌生成时间快9.1倍（0.75秒对6.82秒），解码速度提升1.5倍（29.41对比19.20 tokens/s），支持边缘设备高效部署。模型权重可在HuggingFace获取：https://huggingface.co/NexaAIDev/OmniVLM-968M，推理示例见附录B。
原文摘要 · Abstract (English)
We present OmniVLM, a sub-billion-parameter vision-language model for efficient on-device inference. OmniVLM introduces a token compression mechanism that reduces visual token sequence length from 729 to 81 tokens, significantly reducing computational overhead while preserving visual-semantic fidelity. Through a multi-stage training pipeline of pretraining, supervised fine-tuning, and minimal-edit Direct Preference Optimization (DPO), OmniVLM matches the performance of larger models. On multiple benchmarks including ScienceQA, POPE, and MMMU, OmniVLM outperforms existing baselines like nanoLLAVA within a 968M-parameter footprint. Empirical results on the same laptop demonstrate 9.1x faster time-to-first-token (0.75s vs 6.82s) and 1.5x higher decoding speed (29.41 vs 19.20 tokens/s) compared to nanoLLAVA, enabling efficient deployment on edge devices. The model weights can be accessed on huggingface: https://huggingface.co/NexaAIDev/OmniVLM-968M, and the inference examples can be find in Appendix B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。