arXiv:2409.11315cs.CV2024-09TPAMI被引 9

用脑扫描数据重建带纹理的3D模型,还推出了新数据集。

MinD-3D++: Advancing fMRI-Based 3D Reconstruction with High-Quality Textured Mesh Generation and a Comprehensive Dataset

  • 基于fMRI信号解码生成带细节纹理的3D网格。
  • 在4768个3D物体上实现高语义和空间精度重建。
  • 适合脑科学与视觉生成研究者使用。

从功能性磁共振成像(fMRI)数据中重构3D视觉内容,是认知神经科学与计算机视觉的重要课题。为推进该任务,我们构建了fMRI-3D数据集,涵盖15名受试者,包含4,768个3D物体。该数据集由两部分组成:已发布的fMRI-Shape和本文提出的fMRI-Objaverse。后者包含5名受试者的数据,其中4人也参与了fMRI-Shape。每位受试者观看117类共3,142个3D物体,并附有文本描述,显著提升了数据多样性与应用潜力。我们提出MinD-3D++框架,首次实现从fMRI信号中解码并生成带有精细纹理的3D网格。通过设计语义、结构与纹理层面的评估指标建立新基准。实验表明,MinD-3D++不仅能在语义和空间层面准确重建3D物体,还能深入揭示大脑对3D视觉信息的处理机制。此外,我们分析了所提3D pari fMRI数据集在视觉感兴趣区域(ROIs)中的信号归属。项目页面:https://jianxgao.github.io/MinD-3D。

原文摘要 · Abstract (English)

Reconstructing 3D visuals from functional Magnetic Resonance Imaging (fMRI) data, introduced as Recon3DMind, is of significant interest to both cognitive neuroscience and computer vision. To advance this task, we present the fMRI-3D dataset, which includes data from 15 participants and showcases a total of 4,768 3D objects. The dataset consists of two components: fMRI-Shape, previously introduced and available at https://huggingface.co/datasets/Fudan-fMRI/fMRI-Shape, and fMRI-Objaverse, proposed in this paper and available at https://huggingface.co/datasets/Fudan-fMRI/fMRI-Objaverse. fMRI-Objaverse includes data from 5 subjects, 4 of whom are also part of the core set in fMRI-Shape. Each subject views 3,142 3D objects across 117 categories, all accompanied by text captions. This significantly enhances the diversity and potential applications of the dataset. Moreover, we propose MinD-3D++, a novel framework for decoding textured 3D visual information from fMRI signals. The framework evaluates the feasibility of not only reconstructing 3D objects from the human mind but also generating, for the first time, 3D textured meshes with detailed textures from fMRI data. We establish new benchmarks by designing metrics at the semantic, structural, and textured levels to evaluate model performance. Furthermore, we assess the model's effectiveness in out-of-distribution settings and analyze the attribution of the proposed 3D pari fMRI dataset in visual regions of interest (ROIs) in fMRI signals. Our experiments demonstrate that MinD-3D++ not only reconstructs 3D objects with high semantic and spatial accuracy but also provides deeper insights into how the human brain processes 3D visual information. Project page: https://jianxgao.github.io/MinD-3D.

脑机接口3D生成fMRI纹理重建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。