用小模型生成大量中等质量文本,实现大模型级视觉定位效果
EGM: Efficient Visual Grounding Language Models
- 小模型生成多量中等质量文本,匹配大模型性能
- 在RefCOCO上达91.4 IoU,推理速度提升5.9倍
- 适合资源受限场景下的高效视觉定位任务
视觉定位是视觉语言模型理解真实物理世界的关键能力。以往最先进的视觉语言模型通常模型规模庞大,部署困难且推理缓慢。我们发现,小型与大型视觉语言模型的视觉编码器大小相近,主要差异在于语言模型大小。小型模型在定位能力上的差距源于语言理解能力不足,而非视觉信息处理能力。为此,我们提出高效视觉定位语言模型(EGM):通过生成大量中等质量文本(来自小模型),匹配大模型的少量高质量但昂贵文本。该方法部署友好,显著降低端到端延迟。在RefCOCO基准上,EGM-Qwen3-VL-8B实现91.4 IoU,平均延迟737ms(比Qwen3-VL-235B的4,320ms快5.9倍)。为验证方法普适性,我们引入新的无模态定位设置,要求模型预测物体可见与被遮挡部分。实验表明,该方法持续提升小型模型的常规与无模态定位能力,可达到甚至超越大型模型表现,显著提升视觉定位效率。
原文摘要 · Abstract (English)
Visual grounding is an essential capability of Visual Language Models (VLMs) to understand the real physical world. Previous state-of-the-art grounding visual language models usually have large model sizes, making them heavy for deployment and slow for inference. However, we notice that the sizes of visual encoders are nearly the same for small and large VLMs and the major difference is the sizes of the language models. Small VLMs fall behind larger VLMs in grounding because of the difference in language understanding capability rather than visual information handling. To mitigate the gap, we introduce 'Efficient visual Grounding language Models' (EGM): generate many mid-quality tokens (from small models) to match the performance of large VLMs with few high-quality but expensive tokens. This method is deployment-friendly, and yields better end-to-end latency: On the RefCOCO benchmark, our EGM-Qwen3-VL-8B demonstrates 91.4 IoU with an average of 737ms (5.9x faster) latency while Qwen3-VL-235B demands 4,320ms to reach 90.5 IoU. To validate our approach's generality, we further set up a new amodal grounding setting that requires the model to predict both the visible and occluded parts of the objects. Experiments show our method consistently improves both vanilla and amodal grounding capabilities of small models to match or outperform larger models, thereby improving efficiency for visual grounding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。