arXiv:2607.02083cs.CV2026-07

用自回归语言模型预测眼球扫描路径,实现个性化与任务导向的视觉注意力建模。

DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction

论文配图:DeepGaze3.5-VL: Modeling Scanpaths via Autoregressive Token Prediction
图 1 · 摘自论文原文
  • 将眼动轨迹转为离散序列,利用视觉-语言模型预训练表示进行建模。
  • 在MIT1003上达2.18比特信息增益,比旧模型提升46%。
  • 可灵活控制注视时长等参数,支持虚拟眼动实验与认知机制研究。

理解人类在场景中随时间变化的视觉注意力,对界面设计和认知状态推断具有重要意义。传统眼动扫描路径建模依赖专用架构和手工先验,结构僵化且难以扩展。本文将扫描路径预测纯粹视为离散序列建模问题:通过将坐标映射到文本词汇表,利用预训练视觉-语言模型的表示能力。该框架可吸收多种变量因素——简单提示即可实现全局条件化,如输入观看者身份以捕捉个性化偏差,或加入任务目标如视觉搜索;同时支持整合每阶段注视的持续时间等属性。自回归对齐实现可扩展、精确的每注视对数似然计算,直接对应常用的信息增益(IG)指标。所提模型DeepGaze3.5-VL在多个数据集上达到新基准,在MIT1003上取得2.18比特的IG,相比DeepGaze III提升46%,且该优势在基线使用相同高容量视觉编码器时依然成立。除预测性能外,生成式框架还可用作强大计算工具,实现实验难以完成的受控行为模拟。我们通过调控前扫视注视时长,仅基于数据便复现了已知的眼动生理现象。

原文摘要 · Abstract (English)

Understanding human visual attention on a scene over time has applications in domains such as interface design and inferring cognitive states. Modeling visual scanpaths has historically relied on specialized architectures with hand-crafted priors. While these architectures can model fixation sequences, their rigid structural biases restrict easy extendability and flexible conditioning. For instance, integrating task-specific instructions or adapting to distinct viewer identities requires custom, disjoint architectural additions. We frame scanpath prediction purely as a discrete sequence modeling task. By mapping coordinates into a text vocabulary, we leverage the pretrained representations of Vision-Language Models. This framing absorbs diverse factors of variation: simple prompting allows for global conditioning, such as providing viewer identities to capture personalized biases, or task-specific objectives like visual search. The framework can also integrate per-fixation attributes, such as individual fixation durations, alongside spatial locations. The autoregressive alignment enables the scalable, exact computation of per-fixation log-likelihoods, directly equivalent to the commonly used Information Gain (IG) metric. Our model, DeepGaze3.5-VL, establishes a new state-of-the-art across multiple datasets, achieving 2.18 bits of IG on MIT1003, a 46% improvement over DeepGaze III. This advantage persists even when baselines use identical high-capacity vision encoders. Beyond predictive performance, our generative framework serves as a powerful computational tool for direct behavioral interventions, allowing for controlled in-silico simulations that would be experimentally difficult or impossible to conduct in vivo. We demonstrate this ability by performing controlled interventions on the durations of pre-saccadic fixations, recovering known oculomotor phenomena purely from data.

眼动建模生成模型视觉注意自回归

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。