用多尺度注意力与专家混合模型提升眼神估计精度,支持跨域适应。
GMGaze: MoE-Based Context-Aware Gaze Estimation with CLIP and Multiscale Transformer

- 引入语义原型条件化,早融合全局上下文特征
- 在4个公开数据集上平均角度误差最低达1.44°,跨域性能领先
- 适合需要高精度眼神识别的智能交互系统
眼神估计通常依赖面部外观预测视线方向。现有基于卷积神经网络、Transformer和对比语言图像预训练(CLIP)的方法面临三大挑战:图像特征晚期融合、缺乏因子感知条件化以及难以实用化的容量扩展。为此,我们提出全局条件化多尺度眼神估计(GMGaze),采用多尺度Transformer架构。模型首先引入语义原型条件化,利用四个学习得到的原型库(光照、背景、头部姿态、外观)调节CLIP全局图像嵌入,生成两个互补的上下文偏置全局标记。这些标记与CLIP块标记及CNN标记在第一层进行融合,实现早期统一融合,避免后期合并导致的信息损失。随后,每个标记通过稀疏的专家混合模块,实现条件化计算容量而不均匀增加参数。为增强跨域适应性,引入对抗域适应技术与特征解耦损失,促使两个全局标记保持去相关性。在四个公开基准(MPIIFaceGaze、EYEDIAP、Gaze360、ETH-XGaze)上的实验表明,GMGaze在所有域内设置中均优于此前基线,平均角度误差分别为2.49°、3.22°、10.16°和1.44°;在两种标准跨域迁移任务中亦取得当前最优(SOTA)结果。
原文摘要 · Abstract (English)
Gaze estimation methods commonly use facial appearances to predict the direction of a person gaze. However, previous studies show three major challenges with convolutional neural network (CNN)-based, transformer-based, and contrastive language-image pre-training (CLIP)-based methods, including late fusion of image features, lack of factor-aware conditioning, and impractical capacity scaling. To address these challenges, we propose Globally-conditioned Multi-scale Gaze estimation (GMGaze), which leverages a multi-scale transformer architecture. Specifically, the model first introduces semantic prototype conditioning, which modulates the CLIP global image embedding using four learned prototype banks (i.e., illumination, background, head pose and appearance) to generate two complementary context-biased global tokens. These tokens, along with the CLIP patch and CNN tokens, are fused at the first layer. This early unified fusion prevents information loss common in late-stage merging. Finally, each token passes through sparse Mixture-of-Experts modules, providing conditional computational capacity without uniformly increasing dense parameters. For cross-domain adaptation, we incorporate an adversarial domain adaptation technique with a feature separation loss that encourages the two global tokens to remain de-correlated. Experiments using four public benchmarks (MPIIFaceGaze, EYEDIAP, Gaze360, and ETH-XGaze) show that GMGaze achieves mean angular errors of 2.49$^\circ$, 3.22$^\circ$, 10.16$^\circ$, and 1.44$^\circ$, respectively, outperforming previous baselines in all within-domain settings. In cross-domain evaluations, it provides state-of-the-art (SOTA) results on two standard transfer routes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。