用预训练模型简化眼球注视点预测，效果更优且结构更轻。
Gaze-LLE: Gaze Target Estimation via Large-Scale Learned Encoders
- 用冻结的DINOv2提取场景特征，结合人物位置提示进行注视解码。
- 在多个基准上达到最新最好性能，优于传统复杂手工设计流程。
- 适合做视觉理解与人机交互研究者参考，尤其关注高效模型设计。
我们解决眼球注视点估计问题，即预测一个人在场景中看向何处。该任务需要同时推理人的外观和场景内容。以往方法采用日益复杂的手工设计流水线，精心融合来自独立场景编码器、头部编码器及深度、姿态等辅助信号的特征。受通用特征提取器在多种视觉任务中成功启发，我们提出Gaze-LLE，一种基于Transformer的新框架，通过利用冻结的DINOv2编码器特征，简化了注视点估计流程。对场景提取单一特征表示，并通过人特异性位置提示，由轻量级模块解码注视方向。我们在多个注视基准上验证了其领先性能，并进行了详尽分析以支持设计选择。代码已开源：http://github.com/fkryan/gazelle。
原文摘要 · Abstract (English)
We address the problem of gaze target estimation, which aims to predict where a person is looking in a scene. Predicting a person's gaze target requires reasoning both about the person's appearance and the contents of the scene. Prior works have developed increasingly complex, hand-crafted pipelines for gaze target estimation that carefully fuse features from separate scene encoders, head encoders, and auxiliary models for signals like depth and pose. Motivated by the success of general-purpose feature extractors on a variety of visual tasks, we propose Gaze-LLE, a novel transformer framework that streamlines gaze target estimation by leveraging features from a frozen DINOv2 encoder. We extract a single feature representation for the scene, and apply a person-specific positional prompt to decode gaze with a lightweight module. We demonstrate state-of-the-art performance across several gaze benchmarks and provide extensive analysis to validate our design choices. Our code is available at: http://github.com/fkryan/gazelle .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。