用注视数据筛选更符合用户注意力的视频描述，提升生成文本的相关性。
VEGAS: Human-Aligned Video Caption Evaluation via Gaze
- 基于注视轨迹计算跨模态信息量，评估描述与用户关注点匹配度。
- 无需重训练模型，通过拒绝采样选出更贴合人类注意力的描述。
- 适用于个性化视频理解、教学视频生成等需关注用户焦点的场景。
视觉-语言模型在视频字幕生成上表现优异，但生成的内容往往无法捕捉个体观众的注意力。我们提出 VEGAS（Video caption Evaluation via GAze Score），一种无需训练的评估指标，利用测试时的注视数据采样个性化、注意力对齐的文本。该方法为跨模态信息论度量，量化候选字幕与观察者关注点的匹配程度。为评估 VEGAS，我们构建了一个包含第一人称活动与教学幻灯片的数据集，配以同步的注视与参考标注。随后通过拒绝采样选择字幕，无需模型重训练。实验表明，基于 VEGAS 选取的字幕显著更符合人类注意力，并提升了下游字幕到视频检索性能，证明了推理阶段引入观众注意力的实际价值。
原文摘要 · Abstract (English)
Vision-language models excel at video captioning, yet typically generate descriptions that fail to capture individual viewers' attention. We propose VEGAS (Video caption Evaluation via GAze Score), a training-free metric that leverages test-time gaze to sample personalized, attention-aligned text. It is a cross-modal, information-theoretic metric that quantifies how well a candidate caption matches a viewer's focus. To evaluate VEGAS, we curate a dataset of egocentric activities and instructional slides paired with synchronized gaze and reference annotations. We then select captions based on VEGAS via rejection sampling without model retraining. Experiments show that VEGAS-selected captions align significantly better with human focus and improve downstream caption-to-video retrieval, demonstrating the practical utility of incorporating viewer attention during inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。