用脑电波生成带情绪记忆的音视频,让回忆重现。
Revisiting Your Memory: Reconstruction of Affect-Contextualized Memory via EEG-guided Audiovisual Generation
- 通过脑电波提取情绪动态,指导音视频生成。
- 情绪轨迹解码准确率F1达0.9,生成内容与真实回忆高度匹配。
- 适合个性化媒体创作与神经情绪研究者使用。
本文提出RevisitAffectiveMemory任务,旨在通过脑电图(EEG)提取的情绪信号,生成带有情感语境的视听内容以重建个人记忆。为此,我们构建了EEG-AffectiveMemory数据集,包含九名参与者在回忆记忆时的文本描述、图像、音乐及对应的EEG记录。我们提出三阶段框架RYM,实现同步音视频生成并保持个人情绪动态轨迹。实验表明,该方法可成功从神经信号中解码个体情绪轨迹(F1=0.9),生成内容在定性和定量上均忠实还原记忆情感语境。参与者报告生成内容与回忆情绪高度一致,且基于实际情绪轨迹生成的内容,其情绪相关性(r=0.265, p<.05)和用户偏好度(56%)显著优于随机重排版本。本研究推动了情绪解码与基于神经信号的个性化媒体生成应用。代码与数据集已开源。
原文摘要 · Abstract (English)
In this paper, we introduce RevisitAffectiveMemory, a novel task designed to reconstruct autobiographical memories through audio-visual generation guided by affect extracted from electroencephalogram (EEG) signals. To support this pioneering task, we present the EEG-AffectiveMemory dataset, which encompasses textual descriptions, visuals, music, and EEG recordings collected during memory recall from nine participants. Furthermore, we propose RYM (Revisit Your Memory), a three-stage framework for generating synchronized audio-visual contents while maintaining dynamic personal memory affect trajectories. Experimental results demonstrate our method successfully decodes individual affect dynamics trajectories from neural signals during memory recall (F1=0.9). Also, our approach faithfully reconstructs affect-contextualized audio-visual memory across all subjects, both qualitatively and quantitatively, with participants reporting strong affective concordance between their recalled memories and the generated content. Especially, contents generated from subject-reported affect dynamics showed higher correlation with participants' reported affect dynamics trajectories (r=0.265, p<.05) and received stronger user preference (preference=56%) compared to those generated from randomly reordered affect dynamics. Our approaches advance affect decoding research and its practical applications in personalized media creation via neural-based affect comprehension. Codes and the dataset are available at https://github.com/ioahKwon/Revisiting-Your-Memory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。