无需头部先验信息,统一解决视线目标与视线跟随预测。
GaTector+: A Unified Head-free Framework for Gaze Object and Gaze Following Prediction
- 共享主干+特化模块,分任务提取通用与专用特征。
- 引入头检测分支与基于头部的注意力机制,实现无先验感知。
- 提出新评估指标mSoC,更敏感捕捉框位置差异。
视线目标检测与视线跟随是理解人类视线行为或意图的基础任务。然而,以往方法通常分开处理这两项任务,且在训练和部署阶段均依赖头部相关先验知识,需额外网络提取头部位置,无法实现端到端联合优化,限制了实际应用。为此,我们提出GaTector+,一种统一框架,可在推理时摆脱对头部先验的依赖。该框架采用扩展的特定-通用-特定特征提取结构,共享主干网络同时提取通用特征用于两个任务;在主干前后加入特化模块,以更好适应各子任务特性。为获取头部信息而不依赖先验,我们嵌入一个头部检测分支预测每个人的头部。在回归视线点前,设计基于头部的注意力机制,融合感知特征与视线特征。针对视线热图学习缓慢的问题,提出注意力监督机制加速收敛。最后,提出新的评估指标均候选相似度(mSoC),对边界框差异更敏感。在多个基准数据集上的实验表明,该模型在视线目标检测与视线跟随任务中均表现优异。
原文摘要 · Abstract (English)
Gaze object detection and gaze following are fundamental tasks for interpreting human gaze behavior or intent. However, most previous methods usually solve these two tasks separately, and their prediction of gaze objects and gaze following typically depend on head-related prior knowledge during both the training phase and real-world deployment. This dependency necessitates an auxiliary network to extract head location, thus precluding joint optimization across the entire system and constraining the practical applicability. To this end, we propose GaTector+, a unified framework for gaze object detection and gaze following, which eliminates the dependence on the head-related priors during inference. Specifically, GaTector+ uses an expanded specific-general-specific feature extractor that leverages a shared backbone, which extracts general features for gaze following and object detection using the shared backbone while using specific blocks before and after the shared backbone to better consider the specificity of each sub-task. To obtain head-related knowledge without prior information, we first embed a head detection branch to predict the head of each person. Then, before regressing the gaze point, a head-based attention mechanism is proposed to fuse the sense feature and gaze feature with the help of head location. Since the suboptimization of the gaze point heatmap leads to the performance bottleneck, we propose an attention supervision mechanism to accelerate the learning of the gaze heatmap. Finally, we propose a novel evaluation metric, mean Similarity over Candidates (mSoC), for gaze object detection, which is more sensitive to variations between bounding boxes. The experimental results on multiple benchmark datasets demonstrate the effectiveness of our model in both gaze object detection and gaze following tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。