用视线信号训练视觉语言模型，提升对第一人称行为的预测能力。
Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understanding
- 训练时引入视线注意力正则化，让模型关注与人眼一致区域
- 未来事件预测提升11分，当前活动理解提升约7分
- 适用于辅助机器人、人机协作等真实场景
视线提供了注意力、短期意图和未来动作的重要线索，是建模第一人称行为的有力信号。本文提出一种注视正则化框架，增强视觉语言模型（VLMs）在两个关键第一人称理解任务中的表现：细粒度未来事件预测和当前活动理解。不同于以往仅依赖视觉输入或将视线作为辅助信号的方法，本方法仅在训练阶段使用视线信息。我们设计了一种注视正则化的注意力机制，使模型关注区域与人类视觉注视对齐。该方法灵活且模块化，可适配多种基于注意力的VLM架构。实验表明，相比未使用注视正则化的基线模型，本方法在未来事件预测上提升达11分，在当前活动理解上提升约7分。结果凸显了注视引导训练在提升第一人称VLM准确性和鲁棒性方面的价值。本工作为利用人类注视提升VLM在实际场景（如辅助机器人、人机协作）中预测能力奠定了基础。代码及更多信息见：https://github.com/anupampani/Gaze-VLM
原文摘要 · Abstract (English)
Eye gaze offers valuable cues about attention, short-term intent, and future actions, making it a powerful signal for modeling egocentric behavior. In this work, we propose a gaze-regularized framework that enhances VLMs for two key egocentric understanding tasks: fine-grained future event prediction and current activity understanding. Unlike prior approaches that rely solely on visual inputs or use gaze as an auxiliary input signal , our method uses gaze only during training. We introduce a gaze-regularized attention mechanism that aligns model focus with human visual gaze. This design is flexible and modular, allowing it to generalize across multiple VLM architectures that utilize attention. Experimental results show that our approach improves semantic prediction scores by up to 11 for future event prediction and around 7 for current activity understanding, compared to the corresponding baseline models trained without gaze regularization. These results highlight the value of gaze-guided training in improving the accuracy and robustness of egocentric VLMs. Overall, this work establishes a foundation for using human gaze to enhance the predictive capabilities of VLMs in real-world scenarios like assistive robots and human-machine collaboration. Code and additional information is available at: https://github.com/anupampani/Gaze-VLM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。