用自然语言提示定位视线目标,让眼神分析更灵活准确。
Gaze Target Estimation Anywhere with Concepts

- 用文本或视觉提示直接指定分析对象,端到端完成定位与注视估计。
- 在12万张带提示标注的图像上训练,实测在真实临床数据上表现领先。
- 首次实现可自由提示的视线目标估计,适合需要快速适配新任务的场景。
从自然场景图像中估计人类视线目标是一项重要且具有挑战性的任务。现有方法多依赖脆弱的多阶段流水线,需显式输入头框、人体姿态等信息,易因检测错误导致失败。同时,这些方法缺乏通过自然语言提示灵活指定分析目标的能力。为此,我们提出可提示的视线目标估计(PGE)任务,构建一种基于概念的端到端分析范式。PGE通过文本提示(如“穿红衬衫的男孩”)或坐标提示(如“点[0.52, 0.48]”)指定具体分析对象,融合主体定位与视线估计,摆脱对中间步骤的依赖。我们开发了可扩展的数据引擎,构建了包含12万张高质量图像对的Gaze-Co数据集与基准。同时提出首个专为PGE设计的模型GazeAnywhere,采用基于Transformer的检测器,融合冻结编码器特征,同步完成主体定位、是否在画面内判断及视线热图估计。GazeAnywhere在多个PGE基准上达到当前最优性能,即使在困难的跨领域真实临床数据集上也表现优异。代码已开源于github.com/IrohXu/GazeAnywhere。
原文摘要 · Abstract (English)
Estimating human gaze targets from images in-the-wild is an important and formidable task. Existing approaches primarily employ brittle, multi-stage pipelines that require explicit inputs, like head bounding boxes and human pose, in order to identify the subject of gaze analysis. As a result, detection errors can cascade and lead to failure. Moreover, these prior works lack the flexibility of specifying the gaze analysis task via natural language prompting, an approach which has been shown to have significant benefits in convenience and scalability for other image analysis tasks. To overcome these limitations, we introduce the Promptable Gaze Target Estimation (PGE) task, a new end-to-end, concept-driven paradigm for gaze analysis. PGE conditions gaze prediction on flexible user text or visual prompts (e.g., "the boy in the red shirt" or "person in point [0.52, 0.48]") to identify a specific subject for gaze analysis. This approach integrates subject localization with gaze estimation, and eliminates the rigid dependency on intermediate analysis stages. We develop a scalable data engine to generate Gaze-Co (Gaze Estimation with Concepts), a dataset and benchmark of 120K high-quality, prompt-annotated image pairs. We also propose GazeAnywhere, the first model designed for PGE. GazeAnywhere uses a transformer-based detector to fuse features from frozen encoders and simultaneously solves subject localization, in/out-of-frame presence, and gaze target heatmap estimation. GazeAnywhere achieves state-of-the-art performance on multiple PGE benchmarks, setting a strong baseline for this new problem even on a difficult out-of-domain, real-world clinical dataset. GazeAnywhere is open-sourced in github.com/IrohXu/GazeAnywhere.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。