arXiv:2607.10130cs.CV2026-07中稿 · IJCAI

用文本提示提升视线目标预测精度,无需额外标注。

TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues

论文配图:TextGaze: Prompting Gaze Target Estimation with Textual Scene Cues
图 1 · 摘自论文原文
  • 利用大视觉语言模型提取文本线索,融合视觉与语义信息。
  • 在四个数据集上表现优异,跨数据集泛化能力强。
  • 轻量级设计适合实际应用,尤其适合资源受限场景。

视线目标估计旨在推断人在场景中的注视位置。主流方法中,多分支结构需额外标注,增加负担且影响跨域迁移;而简化设计则过度依赖低层视觉显著性,导致预测注意力与真实注视点不一致。为此,我们提出TextGaze,一种统一的跨模态架构,利用大视觉语言模型(LVLM)作为可扩展的语义引导,平衡两种设计范式。模型从冻结编码器提取视觉特征,并通过LVLM获取与视线对齐的文本线索。设计了具有分层文本监督的Transformer融合模块以保留任务语义,采用轻量级解码头联合预测视线热图及是否在画面内。在四个主流数据集上的实验表明,该方法在关键指标上表现竞争力,且无需额外微调即可实现稳健的跨数据集泛化。整体上,我们提供了一种更简洁的替代方案,并凸显了LVLM作为低成本辅助引导在视线估计中的潜力。

原文摘要 · Abstract (English)

Gaze target estimation aims to infer the position of a person's gaze within a scene. Within mainstream design logic, multi-branch methods require extra supervision and annotations, while streamlined designs prioritize low-level visual saliency over true gaze intent. The former leads to a high annotation burden and hinders domain transfer, whereas the latter causes misalignment between predicted attention and actual gaze targets. To address this issue, we propose TextGaze, a unified cross-modal architecture that leverages a Large Vision-Language Model (LVLM) as scalable semantic guidance to balance the two design paradigms. The model extracts visual features from a frozen encoder and utilizes an LVLM to obtain gaze-aligned textual cues. We design a transformer-based fusion module with hierarchical text supervision to preserve task semantics. Lightweight decoding heads enable the joint prediction of gaze heatmaps and in-/out-of-frame status. We evaluate our method on four mainstream datasets, and the results show competitive performance across key metrics with robust cross-dataset generalisation without extra fine-tuning. Overall, we provide a streamlined alternative to traditional designs and highlight the potential of LVLMs as accessible auxiliary guidance for gaze estimation.

视线估计跨模态大模型应用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。