首个面向脑内图像重建的fMRI基准数据集,助力脑机接口实用化。
NSD-Imagery: A benchmark dataset for extending fMRI vision decoding methods to mental imagery
- 构建人类脑活动与想象图像配对的数据集,填补视觉解码空白。
- 模型在想象图像重建上表现远差于真实图像,且性能与视觉重建无关。
- 简单线性架构比复杂模型更适应脑内图像解码,适合医疗与脑机应用。
我们发布NSD-Imagery,一个与自然场景数据集(NSD)互补的人类fMRI活动与内心图像配对的基准数据集。现有基于NSD训练的模型仅在真实图像重建上评估,而本数据集可评测其在想象图像重建上的表现。由于脑内想象图像信号信噪比和空间分辨率较低,该任务极具挑战性,但对医疗与脑机接口等实际应用至关重要。我们在NSD-Imagery上测试了多款开源模型(MindEye1、MindEye2、Brain Diffuser、iCNN、Takagi et al.),发现其在想象图像重建上的性能与视觉重建性能基本脱钩。进一步分析表明,采用简单线性解码或跨模态特征解码的模型在跨任务泛化上表现更优,而复杂架构易过拟合于视觉数据。结果表明,构建心理图像数据集对实现实际应用至关重要,并确立了NSD-Imagery作为重要研究资源的地位。
原文摘要 · Abstract (English)
We release NSD-Imagery, a benchmark dataset of human fMRI activity paired with mental images, to complement the existing Natural Scenes Dataset (NSD), a large-scale dataset of fMRI activity paired with seen images that enabled unprecedented improvements in fMRI-to-image reconstruction efforts. Recent models trained on NSD have been evaluated only on seen image reconstruction. Using NSD-Imagery, it is possible to assess how well these models perform on mental image reconstruction. This is a challenging generalization requirement because mental images are encoded in human brain activity with relatively lower signal-to-noise and spatial resolution; however, generalization from seen to mental imagery is critical for real-world applications in medical domains and brain-computer interfaces, where the desired information is always internally generated. We provide benchmarks for a suite of recent NSD-trained open-source visual decoding models (MindEye1, MindEye2, Brain Diffuser, iCNN, Takagi et al.) on NSD-Imagery, and show that the performance of decoding methods on mental images is largely decoupled from performance on vision reconstruction. We further demonstrate that architectural choices significantly impact cross-decoding performance: models employing simple linear decoding architectures and multimodal feature decoding generalize better to mental imagery, while complex architectures tend to overfit visual training data. Our findings indicate that mental imagery datasets are critical for the development of practical applications, and establish NSD-Imagery as a useful resource for better aligning visual decoding methods with this goal.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。