arXiv:2601.12534cs.CVcs.AI2026-01被引 1

用自监督方法从低清视频中重建眼动,捕捉情绪信号。

Encoding Emotion Through Self-Supervised Eye Movement Reconstruction

  • 通过自监督眼动重建,从无标注视频提取情绪相关特征。
  • 模型在预测笑声、哭泣、叹气等情绪行为上表现良好。
  • 适合对情绪识别、低质视频分析感兴趣的研究者。

情绪表达与眼动的关系已有充分研究,眼动模式是情绪的可靠指标。但多数研究依赖高精度眼动追踪设备,限制了应用范围。本文利用美国大屠杀纪念基金会视觉历史档案馆中幸存者讲述奥斯维辛集中营经历的视频访谈,探索如何从自然场景下的低分辨率视频中预测多模态情绪标记。受语言模型预训练启发,提出一种新颖的眼动检测模型,采用自监督眼动重建方法,有效利用未标注视频数据。使用该模型的编码嵌入,在两个下游任务上进行微调:一是将眼动与语音中的情绪方向估计对齐;二是以眼动为输入预测三类瞬时情绪行为——笑、哭/抽泣、叹气。结果表明,新模型能有效预测情绪结果,并观察到预训练性能与情绪处理性能呈正相关。结论:自监督眼动重建是编码情绪信号的有效方法。

原文摘要 · Abstract (English)

The relationship between emotional expression and eye movement is well-documented, with literature establishing gaze patterns are reliable indicators of emotion. However, most studies utilize specialized, high-resolution eye-tracking equipment, limiting the potential reach of findings. We investigate how eye movement can be used to predict multimodal markers of emotional expression from naturalistic, low-resolution videos. We utilize a collection of video interviews from the USC Shoah Foundation's Visual History Archive with Holocaust survivors as they recount their experiences in the Auschwitz concentration camp. Inspired by pretraining methods on language models, we develop a novel gaze detection model that uses self-supervised eye movement reconstruction that can effectively leverage unlabeled video. We use this model's encoder embeddings to fine-tune models on two downstream tasks related to emotional expression. The first is aligning eye movement with directional emotion estimates from speech. The second task is using eye gaze as a predictor of three momentary manifestations of emotional behaviors: laughing, crying/sobbing, and sighing. We find our new model is predictive of emotion outcomes and observe a positive correlation between pretraining performance and emotion processing performance for both experiments. We conclude self-supervised eye movement reconstruction is an effective method for encoding the affective signal they carry.

情绪识别自监督学习眼动分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。