无需标注数据,多视角联合估计3D注视点,精度领先。
Self-Supervised Multi-View 3D Gaze Target Estimation via Probabilistic Ray Marching

- 用概率射线追踪融合多视角信息,直接预测3D注视点。
- 在MVGT数据集上超越全监督基线,实现当前最佳性能。
- 适合无标注场景下的3D注视估计研究者使用。
我们提出一种自监督方法Self-MVGTE,从多视角图像中估计3D注视目标。不同于以往各视角独立估计2D注视点的方法,Self-MVGTE首次直接在3D空间中预测注视目标。该方法不依赖目标场景的真实标注,仅需校准后的多视角输入图像、单目注视点估计模型生成的伪2D标签,以及单目3D注视向量估计模型提供的3D注视向量。核心挑战在于这些伪标签存在固有噪声和多视角不一致问题。为此,我们设计了一种概率射线追踪框架,建模伪标签不确定性,并利用3D注视向量作为几何先验。具体而言,先将3D注视向量融入单目注视点估计模型以提升其在未见场景中的泛化能力,生成更高质量的伪标签;随后,通过从眼位置沿注视向量周围投射一组射线,构建3D注视锥,严格约束解空间;在该锥内,采用现成的DINOv2与Depth-Anything-3模型,提出深度引导特征采样策略,估计注视目标的空间似然分布;最后,将伪标签转换为目标分布,并进行软优化。在MVGT数据集上的大量实验表明,Self-MVGTE达到当前最优性能,优于现有全监督基线。
原文摘要 · Abstract (English)
We present a self-supervised approach, Self-MVGTE, for estimating 3D gaze targets from multiple camera views. Unlike existing methods that independently estimate 2D gaze targets per camera view, Self-MVGTE predicts gaze targets directly in 3D space for the first time. Moreover, it does not require any ground-truth annotations from the target scene and uses only the multi-view input images from a calibrated camera setup, pseudo 2D gaze target labels from a monocular gaze target estimation model, and 3D gaze vectors from a monocular 3D gaze estimation model. A key challenge is that these pseudo labels are inherently noisy and multi-view inconsistent. To address this, we propose a probabilistic ray marching framework, which models the uncertainty of these pseudo labels and exploits 3D gaze vectors as geometric priors. Specifically, these gaze vectors are first integrated into the monocular gaze target estimation model to improve its generalization to unseen scenes, producing higher-quality pseudo labels. Then, for 3D gaze target estimation, we construct a 3D gaze cone by casting a bundle of rays from the eye position around the gaze vector to strictly constrain the solution space. Within this cone, we propose a depth-guided feature sampling strategy using off-the-shelf DINOv2 and Depth-Anything-3 models, and estimate a spatial likelihood distribution of the gaze target. Finally, we convert the pseudo gaze target labels into a target distribution and softly optimize the network. Extensive experiments on the MVGT dataset show that Self-MVGTE achieves state-of-the-art performance, surpassing existing fully-supervised baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。