让大模型学会关注视线，提升视频理解准确率
GazeQwen: Lightweight Gaze-Conditioned LLM Modulation for Streaming Video Understanding
- 通过轻量级模块在特定层注入视线信息
- 在流式注视基准上达63.9%准确率，提升16.1点
- 适合需要高效视觉注意力建模的研究者
当前多模态大语言模型无法有效利用眼动信息进行视频理解，即使有视线提示。我们提出GazeQwen，一种参数高效的方案，通过隐藏状态调制为开源多模态大模型注入视线感知能力。核心是一个仅含1-500万可训练参数的紧凑视线重采样器，将V-JEPA 2.1视频特征与基于注视的定位编码结合，生成加性残差，并通过前向钩子注入选定的LLM解码器层。可选的第二阶段训练加入低秩适配器（LoRA）以实现更紧密集成。在StreamGaze基准全部10个任务上，GazeQwen达到63.9%准确率，较相同Qwen2.5-VL-7B基线提升16.1点，优于GPT-4o（+10.5点），是所有开源及专有模型中的最高分。结果表明，学习在何处注入视线信息比扩大模型规模或优化提示更有效。代码与模型权重已公开于https://github.com/phamtrongthang123/gazeqwen。
原文摘要 · Abstract (English)
Current multimodal large language models (MLLMs) cannot effectively utilize eye-gaze information for video understanding, even when gaze cues are supplied via visual overlays or text descriptions. We introduce GazeQwen, a parameter efficient approach that equips an open-source MLLM with gaze awareness through hidden-state modulation. At its core is a compact gaze resampler (~1-5 M trainable parameters) that encodes V-JEPA 2.1 video features together with fixation-derived positional encodings and produces additive residuals injected into selected LLM decoder layers via forward hooks. An optional second training stage adds low-rank adapters (LoRA) to the LLM for tighter integration. Evaluated on all 10 tasks of the StreamGaze benchmark, GazeQwen reaches 63.9% accuracy, a +16.1 point gain over the same Qwen2.5-VL-7B backbone with gaze as visual prompts and +10.5 points over GPT-4o, the highest score among all open-source and proprietary models tested. These results suggest that learning where to inject gaze within an LLM is more effective than scaling model size or engineering better prompts. All code and checkpoints are available at https://github.com/phamtrongthang123/gazeqwen .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。