提升视觉大模型的凝视推理能力，让系统更准确跟随非显著目标的视线。
Enhancing Gaze Reasoning in Vision Foundation Models for Gaze Following
- 通过局部LoRA适配头部位特征，增强凝视线索学习。
- 引入锥外惩罚机制，使头部特征与场景对齐并强化凝视信号。
- 在非显著目标上性能显著提升，适合追求精准凝视追踪的研究者。
凝视跟随需要同时具备场景理解与凝视推理能力以定位画面中人物注视的目标。近期视觉基础模型（VFMs）在该任务上表现优异，简化了架构设计并超越了先前方法。然而我们发现，基于VFMs的方法存在关键缺陷：虽然大幅提升了场景理解能力，但对凝视推理的贡献有限。导致现有方法往往依赖语义显著物体而非真实凝视线索，当目标不显著时性能下降。为此，我们提出一种新型训练机制以增强VFMs在凝视跟随中的推理能力。方法包括：(1) 头部条件局部LoRA，实现局部适配，在保留场景标记学习的同时改善头部标记的凝视推理学习；(2) 锥外惩罚，将凝视线索注入头部标记并使其与场景标记对齐。在GazeFollow和VAT数据集上的实验表明，本方法达到当前最优性能，尤其在目标非语义显著时提升明显。研究结果为未来凝视跟随研究提供了重要启示。代码将在论文被接收后公开。
原文摘要 · Abstract (English)
Gaze following requires both scene understanding and gaze reasoning to localize the gaze target of an in-scene person. Recently, vision foundation models (VFMs) have demonstrated strong performance on this task, enabling simpler architectures while outperforming prior methods. However, we observe a key limitation of VFM-based approaches: while VFMs substantially improve scene understanding, they contribute little to gaze reasoning. As a result, existing methods often rely on semantically salient objects rather than true gaze cues, leading to degraded performance when targets are not salient. To address this, we propose a novel training mechanism to enhance gaze reasoning in VFMs for gaze following. Our method includes: (1) a head-conditioned local LoRA, which enables localized adaptation to preserve scene token learning while improving head token learning for gaze reasoning; and (2) an out-of-cone penalty, which injects gaze cues into head tokens while aligning them with scene tokens. Experiments on the GazeFollow and VAT datasets demonstrate that our method achieves state-of-the-art performance, with particularly strong improvements when gaze targets are not semantically salient. Our findings offer valuable insights for advancing future gaze following research. We will release the code once the paper is accepted.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。