arXiv:2506.07627cs.CV2025-06被引 6

用动态事件数据引导视觉计算,让模型更省力地理解图像。

Event-Priori-Based Vision-Language Model for Efficient Visual Understanding

  • 通过事件流指导视觉输入稀疏化,只处理重要区域。
  • 在RealWorldQA上实现50%算力节省,准确率保留98%。
  • 适合部署在手机、摄像头等边缘设备的高效视觉模型。

基于大语言模型的视觉语言模型虽显著提升了视觉理解能力,但其高计算开销限制了在资源受限边缘设备上的部署。主要瓶颈在于模型需处理密集且冗余的视觉信息,其中大量内容与文本语义无关,造成无效计算。本文提出一种基于事件先验的视觉语言模型(EP-VLM),核心是利用动态事件视觉中的运动先验,提升模型效率。受人类视觉认知启发,EP-VLM首先用事件数据引导对RGB图像的逐块稀疏化,逐步聚焦于视觉输入中的显著区域;随后设计一种位置保持的标记化策略,使视觉编码器能有效处理事件引导的非结构化稀疏输入,同时保持精确的位置感知。实验表明,与Qwen2-VL系列基线相比,EP-VLM在保持近乎无损准确率的同时实现显著效率提升。例如,在RealWorldQA数据集上,相较于原始Qwen2-VL-2B模型,其减少50%的浮点运算量(FLOPs),同时保留98%的原始准确率。该工作展示了事件视觉先验在提升视觉语言模型推理效率方面的潜力,为实现可持续边缘视觉理解提供了新路径。

原文摘要 · Abstract (English)

Large Language Model (LLM)-based Vision-Language Models (VLMs) have substantially extended the boundaries of visual understanding capabilities. However, their high computational demands hinder deployment on resource-constrained edge devices. A key source of inefficiency stems from the VLM's need to process dense and redundant visual information. Visual inputs contain significant regions irrelevant to text semantics, rendering the associated computations ineffective for inference. This paper introduces a novel Event-Priori-Based Vision-Language Model, termed EP-VLM. Its core contribution is a novel mechanism leveraging motion priors derived from dynamic event vision to enhance VLM efficiency. Inspired by human visual cognition, EP-VLM first employs event data to guide the patch-wise sparsification of RGB visual inputs, progressively concentrating VLM computation on salient regions of the visual input. Subsequently, we construct a position-preserving tokenization strategy for the visual encoder within the VLM architecture. This strategy processes the event-guided, unstructured, sparse visual input while accurately preserving positional understanding within the visual input. Experimental results demonstrate that EP-VLM achieves significant efficiency improvements while maintaining nearly lossless accuracy compared to baseline models from the Qwen2-VL series. For instance, against the original Qwen2-VL-2B, EP-VLM achieves 50% FLOPs savings while retaining 98% of the original accuracy on the RealWorldQA dataset. This work demonstrates the potential of event-based vision priors for improving VLM inference efficiency, paving the way for creating more efficient and deployable VLMs for sustainable visual understanding at the edge.

视觉语言模型事件视觉边缘计算稀疏计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。