用Transformer融合RGBD图像特征，提升眼动追踪精度。
RGBD Gaze Tracking Using Transformer for Feature Fusion
- 采用Transformer融合彩色与深度图像特征
- 无预训练GAN时误差降至26.9mm，优于原模型
- 适合实时眼动追踪系统开发人员参考
本论文实现了一种基于AI的眼动追踪系统，使用包含颜色（RGB）和深度信息的RGBD图像。为融合两类图像提取的特征，采用基于Transformer架构的模块，该组合尚未被充分研究。为此，构建了一个新数据集用于训练模型，因现有数据集要么缺乏深度信息，要么仅提供不适合眼动角度估计的注视点标签。在三个不同数据集上训练、验证并评估多种模型配置。最终模型将用于实时估算用户面对屏幕时的注视方向与注视点。所用模型架构源自Lian等人的工作，其使用生成对抗网络（GAN）同时去除深度图伪影并提取头部姿态特征，在ShanghaiTechGaze+数据集上达到38.7mm的均方欧氏误差。本文提出的带Transformer融合模块的模型在相同数据集上误差为55.3mm，但若不使用预训练GAN模块，误差可降至30.1mm；替换Transformer为多层感知机（MLP）后，误差进一步降至26.9mm。这些结果在其他两个数据集上也一致。在ETH-XGaze数据集上，带Transformer的模型平均角度误差为3.59°，无Transformer为3.26°，而原作者张等人使用的不同架构达到2.04°。
原文摘要 · Abstract (English)
Subject of this thesis is the implementation of an AI-based Gaze Tracking system using RGBD images that contain both color (RGB) and depth (D) information. To fuse the features extracted from the images, a module based on the Transformer architecture is used. The combination of RGBD input images and Transformers was chosen because it has not yet been investigated. Furthermore, a new dataset is created for training the AI models as existing datasets either do not contain depth information or only contain labels for Gaze Point Estimation that are not suitable for the task of Gaze Angle Estimation. Various model configurations are trained, validated and evaluated on a total of three different datasets. The trained models are then to be used in a real-time pipeline to estimate the gaze direction and thus the gaze point of a person in front of a computer screen. The AI model architecture used in this thesis is based on an earlier work by Lian et al. It uses a Generative Adversarial Network (GAN) to simultaneously remove depth map artifacts and extract head pose features. Lian et al. achieve a mean Euclidean error of 38.7mm on their own dataset ShanghaiTechGaze+. In this thesis, a model architecture with a Transformer module for feature fusion achieves a mean Euclidean error of 55.3mm on the same dataset, but we show that using no pre-trained GAN module leads to a mean Euclidean error of 30.1mm. Replacing the Transformer module with a Multilayer Perceptron (MLP) improves the error to 26.9mm. These results are coherent with the ones on the other two datasets. On the ETH-XGaze dataset, the model with Transformer module achieves a mean angular error of 3.59° and without Transformer module 3.26°, whereas the fundamentally different model architecture used by the dataset authors Zhang et al. achieves a mean angular error of 2.04°. On the OTH-Gaze-Estimation dataset created for...
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。