用脑电与核磁同步数据重建观看剧集时的动态画面
CineBrain: A Large-Scale Multi-Modal Brain Dataset During Naturalistic Audiovisual Narrative Processing
- 融合fMRI与EEG信号，构建多模态脑信号解码框架
- 在6小时《生活大爆炸》片段上实现当前最佳视频重建效果
- 揭示听觉皮层激活能提升视觉感知解码精度，适合脑机接口研究者
现有脑信号转图像的研究多聚焦视觉内容，忽视了听觉与视觉的自然整合。为此，我们提出从自然音频-视觉刺激下的多模态脑信号中重建连续视频的新任务。为此，我们推出了首个同步记录fMRI与EEG的大规模数据集CineBrain，包含六小时《生活大爆炸》剧集片段，用于跨模态对齐。我们还首次系统探索了结合fMRI与EEG进行视频重建的方法，并提出CineSync框架，采用多模态融合编码器与神经隐空间解码器，利用两者互补优势显著提升视觉还原质量。分析表明，听觉皮层激活可增强解码准确性，凸显听觉输入在视觉感知中的关键作用。
原文摘要 · Abstract (English)
Most research decoding brain signals into images, often using them as priors for generative models, has focused only on visual content. This overlooks the brain's natural ability to integrate auditory and visual information, for instance, sound strongly influences how we perceive visual scenes. To investigate this, we propose a new task of reconstructing continuous video stimuli from multimodal brain signals recorded during audiovisual stimulation. To enable this, we introduce CineBrain, the first large-scale dataset that synchronizes fMRI and EEG during audiovisual viewing, featuring six hours of \textit{The Big Bang Theory} episodes for cross-modal alignment. We also conduct the first systematic exploration of combining fMRI and EEG for video reconstruction and present CineSync, a framework for reconstructing dynamic video using a Multi-Modal Fusion Encoder and a Neural Latent Decoder. CineSync achieves state-of-the-art performance in dynamic reconstruction, leveraging the complementary strengths of fMRI and EEG to improve visual fidelity. Our analysis shows that auditory cortical activations enhance decoding accuracy, highlighting the role of auditory input in visual perception. Project Page: https://jianxgao.github.io/CineBrain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。