用fMRI信号生成图像描述，避免数据污染，效率更高。
Decoding fMRI Data into Captions using Prefix Language Modeling
- 用DINOv2嵌入作为桥梁，将脑信号映射到图像特征空间。
- 相比传统方法，生成的描述在Flickr30K上达到18.7的BLEU-4得分。
- 适合脑机接口、神经科学领域研究者阅读。
随着大语言模型和潜在扩散模型的发展，脑信号解码近年来取得显著进展。现有基于NSD数据集的研究利用CLIP模型的嵌入进行图像重建，以及GIT模型生成描述，但后者因训练数据包含COCO数据集，存在潜在数据泄露风险。本文提出新方法：从fMRI信号预测DINOv2模型对图像的嵌入，并将该嵌入的[CLS] token作为前缀输入GPT-2模型，大幅降低计算开销。此外，不同于常用的线性回归，本文采用3D卷积神经网络映射脑信号至图像嵌入空间，更好保留体素的空间位置信息。
原文摘要 · Abstract (English)
With the advancements in Large Language and Latent Diffusion models, brain decoding has achieved remarkable results in recent years. The works on the NSD dataset, with stimuli images from the COCO dataset, leverage the embeddings from the CLIP model for image reconstruction and GIT for captioning. However, the current captioning approach introduces the challenge of potential data contamination given that the GIT model was trained on the COCO dataset. In this work, we present an alternative method for decoding brain signals into image captions by predicting a DINOv2 model's embedding of an image from the corresponding fMRI signal and then providing its [CLS] token as the prefix to the GPT-2 language model which decreases computational requirements considerably. Additionally, instead of commonly used Linear Regression, we explore 3D Convolutional Neural Network mapping of fMRI signals to image embedding space for better accounting positional information of voxels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。