用人眼注视点指导视觉计算,大幅降低模型开销却保持高精度。
Eye Gaze Tells You Where to Compute: Gaze-Driven Efficient VLMs
- 利用人眼注视点定位关键区域,动态裁剪冗余视觉特征。
- 在VOILA-COCO上减少93.1%视觉令牌,计算量下降50%仍保高质量。
- 无需训练或修改结构,适合部署在AR/VR等边缘设备上。
视觉语言模型在理解视觉内容方面表现优异,但视觉令牌冗余导致推理效率下降,限制了其在AR/VR等边缘设备上的实时应用。现有方法多依赖学习到的显著性、稀疏注意力或控制器策略进行令牌裁剪,常需架构改动或中间激活访问,引入额外计算与内存开销,并存在提示与关注区域错位问题。本文提出GazeVLM,一种无需训练的框架,以人类眼动作为自然监督信号,引导计算聚焦于关注区域。通过提取眼动驱动的关键区域(ROIs),并可选结合低分辨率全局视图,模拟视网膜中心-周边感知机制,在保留任务相关细节的同时减少冗余令牌。在Qwen2.5-VL-3B/7B与VOILA-COCO数据集上评估,使用GPT-4o进行答案质量打分(覆盖度、准确率、细节、流畅性加权),结果显示GazeVLM最多减少93.1%视觉令牌、59.6%总令牌和50% FLOPs,且优于全分辨率基线模型的答案质量。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) deliver impressive performance in understanding visual content with language instructions. However, redundancy in vision tokens results in the degenerated inference efficiency of VLMs, which hinders real-time use on edge consumer devices such as AR/VR devices. Existing efficiency methods commonly prune visual tokens using learned saliency, sparse attention schedules, or controller policies, but they often require architectural modification or access to intermediate activations. These pipelines add inference-time modules that increase compute and memory and often lead to an accuracy trade-off. Moreover, they also suffer from misalignment between the prompts and the region of interest in the images. Without human guidance, the model may focus on the wrong regions and miss small, high-frequency details when prompts or scenes change. In this paper, we propose GazeVLM, a training-free framework that uses the human eye gaze as a natural supervisory signal to allocate computation where it matters. By extracting gaze-driven regions of interest (ROIs) and optionally combining them with a low-resolution global view, GazeVLM mimics fovea-periphery perception to cut redundant visual tokens while preserving task-relevant details. We evaluate the visual question answering tasks on Qwen2.5-VL-3B/7B on the VOILA-COCO benchmark with human gaze. Quality of the answer is assessed by GPT-4o pairwise judging and a weighted score over coverage, accuracy, details, and fluency. Efficiency is measured by token counts and FLOPs. GazeVLM reduces visual tokens by up to 93.1%, total tokens by up to 59.6%, and FLOPs by 50%, while keeping better answer quality relative to full-resolution baselines. Our results show that aligning model computation with human gaze offers a simple, plug-and-play path toward efficient VLM inference on consumer devices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。