arXiv:2503.11167cs.CVcs.AI2025-03ICCV被引 8

模仿人脑视觉皮层,用fMRI重建视频更清晰可解释。

Neurons: Emulating the Human Visual Cortex Improves Fidelity and Interpretability in fMRI-to-Video Reconstruction

  • 分四步模拟视觉皮层:分物体、认概念、描述场景、生成模糊视频。
  • 视频一致性提升26.6%,语义准确率提高19.1%。
  • 适合脑机接口与临床神经研究,结果可解释性强。

从神经活动解码视觉刺激对理解人类大脑至关重要。尽管fMRI已成功重建静态图像,但因需捕捉运动和场景切换等时空动态,实现fMRI到视频的重建仍具挑战。现有方法虽提升了语义与感知一致性,却难以融合粗粒度fMRI数据与精细视觉特征。受视觉系统层级结构启发,我们提出NEURONS框架,将学习分解为四个相关子任务:关键物体分割、概念识别、场景描述与模糊视频重建。该设计模拟视觉皮层的功能特化,使模型能捕捉多样视频内容。推理阶段,NEURONS为预训练文本到视频扩散模型生成强条件信号以完成视频重建。大量实验表明,NEURONS超越当前最优基线,在视频一致性上提升26.6%,语义级准确率提升19.1%。值得注意的是,NEURONS与视觉皮层具有显著功能相关性,凸显其在脑机接口与临床应用中的潜力。代码与模型权重见:https://github.com/xmed-lab/NEURONS。

原文摘要 · Abstract (English)

Decoding visual stimuli from neural activity is essential for understanding the human brain. While fMRI methods have successfully reconstructed static images, fMRI-to-video reconstruction faces challenges due to the need for capturing spatiotemporal dynamics like motion and scene transitions. Recent approaches have improved semantic and perceptual alignment but struggle to integrate coarse fMRI data with detailed visual features. Inspired by the hierarchical organization of the visual system, we propose NEURONS, a novel framework that decouples learning into four correlated sub-tasks: key object segmentation, concept recognition, scene description, and blurry video reconstruction. This approach simulates the visual cortex's functional specialization, allowing the model to capture diverse video content. In the inference stage, NEURONS generates robust conditioning signals for a pre-trained text-to-video diffusion model to reconstruct the videos. Extensive experiments demonstrate that NEURONS outperforms state-of-the-art baselines, achieving solid improvements in video consistency (26.6%) and semantic-level accuracy (19.1%). Notably, NEURONS shows a strong functional correlation with the visual cortex, highlighting its potential for brain-computer interfaces and clinical applications. Code and model weights are available at: https://github.com/xmed-lab/NEURONS.

fMRI视频生成脑机接口可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。