arXiv:2605.26399cs.CV2026-05

统一视觉语言框架,实现多人精准眼神追踪与语义理解。

OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following

论文配图:OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following
图 1 · 摘自论文原文
  • 双分支结构:语言分支推理语义,空间分支直接提取密集特征
  • 多人群体建模,支持一次性完成多人眼神定位与社会情境推断
  • 基于大模型的高精度连续空间预测,优于传统文本生成方式

理解人类眼神行为对复杂场景理解与人机交互至关重要。传统眼神追随模型仅限于纯空间定位,缺乏对语义目标或复杂社会情境的推理能力,且通常逐人处理,重复计算同一场景图像。尽管近期视觉语言模型(VLMs)具备强大的语义推理能力,但其依赖离散文本生成的特性,限制了在连续空间任务(如眼神定位)中的精度。为此,我们提出OmniGF,一种统一的视觉语言框架,将基础VLM适配于可扩展的多人眼神推理。模型采用双分支解码策略:结构化语言分支生成离散推理状态,连续空间分支直接利用VLM的密集隐藏状态。通过高分辨率眼神目标热图监督提取表征,有效克服仅靠文本生成的坐标瓶颈。此外,为显式建模多人场景,输入中加入从裁剪头像编码的头部嵌入,同时提供所有个体的外观与朝向细粒度线索。通过联合建模所有人并利用VLM强语义能力,OmniGF无缝集成精确空间定位、语义眼神预测与复杂社会眼神推理。大量实验表明,该框架在多个标准基准上达到新最优性能。代码已开源:https://github.com/cvlab-stonybrook/omnigf。

原文摘要 · Abstract (English)

Understanding human gaze behavior is essential for complex scene comprehension and human-computer interaction. Traditional gaze following models are typically restricted to pure spatial localization, lacking the high-level capacity to reason about semantic targets or complex social contexts. Furthermore, these models often process individuals sequentially, requiring redundant computations over the same scene image for multi-person inference. While recent Vision-Language Models (VLMs) offer the exceptional semantic reasoning needed to address gaze-related semantic tasks, their reliance on discrete text generation inherently limits precision in continuous spatial tasks like gaze localization. To bridge this gap, we propose OmniGF, a unified vision-language framework that adapts foundational VLMs for highly scalable multi-person gaze reasoning. The model adopts a dual-branch decoding strategy: a structured language branch generates discrete reasoning states, while a continuous spatial branch directly taps into the VLM's dense hidden states. Supervising these extracted representations with high-resolution gaze target heatmaps effectively overcomes the spatial bottleneck of text-only coordinate generation. Furthermore, to explicitly ground the model in multi-person scenes, we augment the input with head embeddings encoded from cropped head images, providing fine-grained appearance and orientation cues for all individuals simultaneously. By modeling all individuals and leveraging the strong semantic capability of VLMs, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning. Extensive experiments demonstrate that our framework establishes new state-of-the-art performance across multiple standard benchmarks. Code is available at https://github.com/cvlab-stonybrook/omnigf.

眼神追踪视觉语言模型多人群体连续定位

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。