用专家混合模型精准识别视线目标，提升机器人理解人类注意力的能力。
GazeMoE: Perception of Gaze Target with Mixture-of-Experts
- 引入专家混合机制，动态选择视觉线索进行视线目标定位
- 在多个数据集上超越现有方法，出框与入框识别准确率均领先
- 适合需要高精度视线感知的机器人交互与人机协作场景
从可见图像中估计人类视线目标是机器人理解人类注意力的关键任务，但通用神经架构与训练范式的发展仍具挑战。尽管预训练视觉基础模型为定位视线目标提供了前景，但整合眼动、头部姿态、手势和上下文特征等多模态线索，需要自适应且高效的解码机制。受大型视觉-语言模型中专家混合（MoE）思想启发，我们提出 GazeMoE，一种端到端框架，通过 MoE 模块从冻结的基础模型中选择性地利用与视线目标相关的特征。为解决视线目标分类中的类别不平衡问题（在框内 vs. 在框外），并增强鲁棒性，GazeMoE 结合了类别平衡辅助损失及策略性数据增强，包括区域特定裁剪和光照变换。在基准数据集上的大量实验表明，GazeMoE 在具有挑战性的视线估计任务中达到当前最优性能。代码与预训练模型已开源至 https://huggingface.co/zdai257/GazeMoE。
原文摘要 · Abstract (English)
Estimating human gaze target from visible images is a critical task for robots to understand human attention, yet the development of generalizable neural architectures and training paradigms remains challenging. While recent advances in pre-trained vision foundation models offer promising avenues for locating gaze targets, the integration of multi-modal cues -- including eyes, head poses, gestures, and contextual features -- demands adaptive and efficient decoding mechanisms. Inspired by Mixture-of-Experts (MoE) for adaptive domain expertise in large vision-language models, we propose GazeMoE, a novel end-to-end framework that selectively leverages gaze-target-related cues from a frozen foundation model through MoE modules. To address class imbalance in gaze target classification (in-frame vs. out-of-frame) and enhance robustness, GazeMoE incorporates a class-balancing auxiliary loss alongside strategic data augmentations, including region-specific cropping and photometric transformations. Extensive experiments on benchmark datasets demonstrate that our GazeMoE achieves state-of-the-art performance, outperforming existing methods on challenging gaze estimation tasks. The code and pre-trained models are released at https://huggingface.co/zdai257/GazeMoE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。