用视觉语言模型增强眼神估计，精度显著提升。
GazeFormer-MoE: Context-Aware Gaze Estimation via CLIP and MoE Transformer
- 用可学习原型库融合光照、姿态等语义信息，增强全局特征
- 在多个数据集上达到2.49°~10.16°的最低误差，最高提升64%
- 适合做高精度眼神追踪的研究者和工业应用开发者
我们提出一种语义调制的多尺度Transformer用于3D眼神估计。模型将CLIP全局特征与可学习的原型库（光照、头部姿态、背景、方向）结合，将这些增强后的全局向量与CLIP patch tokens及高分辨率CNN tokens在统一注意力空间中融合，并用路由/共享的Mixture of Experts（MoE）替换部分前馈网络块以提升条件建模能力。在MPIIFaceGaze、EYEDIAP、Gaze360和ETH-XGaze数据集上的评估显示，模型分别达到2.49°、3.22°、10.16°和1.44°的角误差，相比之前结果最高提升64%。消融实验表明性能提升主要来自原型调制、跨尺度融合、MoE机制及超参数优化。代码已公开于https://github.com/AIPMLab/Gazeformer。
原文摘要 · Abstract (English)
We present a semantics modulated, multi scale Transformer for 3D gaze estimation. Our model conditions CLIP global features with learnable prototype banks (illumination, head pose, background, direction), fuses these prototype-enriched global vectors with CLIP patch tokens and high-resolution CNN tokens in a unified attention space, and replaces several FFN blocks with routed/shared Mixture of Experts to increase conditional capacity. Evaluated on MPIIFaceGaze, EYEDIAP, Gaze360 and ETH-XGaze, our model achieves new state of the art angular errors of 2.49°, 3.22°, 10.16°, and 1.44°, demonstrating up to a 64% relative improvement over previously reported results. ablations attribute gains to prototype conditioning, cross scale fusion, MoE and hyperparameter. Our code is publicly available at https://github. com/AIPMLab/Gazeformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。