融合多数据集提升眼神估计精度,解决标注不一致与姿态差异问题。
Merging Multiple Datasets for Improved Appearance-Based Gaze Estimation
- 采用两阶段注意力融合结构,分步整合双眼与面部特征。
- 引入眼神自适应模块,修复不同数据集间的标注偏差,提升泛化能力。
- 在多个基准上实现10%-20%性能提升,适合跨数据集眼神估计研究者。
为训练和测试基于外观的眼神估计器,已创建多个数据集。直观上,更多数据应带来更好性能,但将多个数据集合并训练单一估计器时,性能常未提升。原因之一是实验协议差异导致头姿、眼神角度、光照等分布不一致;另一原因是不同方法定义眼神角度造成标签不匹配。本文提出两项创新:一是改进估计器架构,采用两阶段变压器融合(TTGF)方法,分别融合双眼与面部信息,再跨双眼整合,增强对头姿变化的鲁棒性;二是引入眼神自适应模块(GAM),为每个数据集独立校正共享估计器的输出,解决标注不一致问题。实验表明,两项方法单独及联合使用均显著优于当前最优(提升10%–20%)。代码已开源:https://github.com/HKUST-NISL/GazeSetMerge。
原文摘要 · Abstract (English)
Multiple datasets have been created for training and testing appearance-based gaze estimators. Intuitively, more data should lead to better performance. However, combining datasets to train a single esti-mator rarely improves gaze estimation performance. One reason may be differences in the experimental protocols used to obtain the gaze sam-ples, resulting in differences in the distributions of head poses, gaze an-gles, illumination, etc. Another reason may be the inconsistency between methods used to define gaze angles (label mismatch). We propose two innovations to improve the performance of gaze estimation by leveraging multiple datasets, a change in the estimator architecture and the intro-duction of a gaze adaptation module. Most state-of-the-art estimators merge information extracted from images of the two eyes and the entire face either in parallel or combine information from the eyes first then with the face. Our proposed Two-stage Transformer-based Gaze-feature Fusion (TTGF) method uses transformers to merge information from each eye and the face separately and then merge across the two eyes. We argue that this improves head pose invariance since changes in head pose affect left and right eye images in different ways. Our proposed Gaze Adaptation Module (GAM) method handles annotation inconsis-tency by applying a Gaze Adaption Module for each dataset to correct gaze estimates from a single shared estimator. This enables us to combine information across datasets despite differences in labeling. Our experi-ments show that these innovations improve gaze estimation performance over the SOTA both individually and collectively (by 10% - 20%). Our code is available at https://github.com/HKUST-NISL/GazeSetMerge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。