研究脑电图等设备如何影响图像解码,发现数据量决定性能提升。
Scaling laws for decoding images from brain activity
- 对比四种非侵入式脑成像设备,评估模型在单次试次下的实时解码能力。
- 解码性能随训练数据量呈对数线性增长,无明显上限。
- 增加数据量比增加受试者数量更能提升解码效果,适合真实场景应用。
生成式AI推动了从脑活动解码图像的发展。本文系统比较了四种非侵入式设备:脑电图(EEG)、脑磁图(MEG)、高场强功能磁共振(3T fMRI)和超高场强(7T fMRI)。基于迄今为止最大的基准测试——涵盖8个公开数据集、84名志愿者、498小时脑记录和230万次自然图像的脑响应——评估解码模型。不同于以往研究,聚焦于单次试次解码以模拟实时场景。主要发现:第一,在训练集规模相近时,精度更高的神经影像设备表现更优;但深度学习相比线性模型在噪声更大的设备上提升更显著。第二,解码性能随训练数据量持续提升,未出现平台期,呈现对数线性增长趋势。第三,性能提升主要依赖每位受试者的数据量,增加受试者数量带来的增益有限。这些结果为非侵入式脑信号图像解码的规模化路径提供了明确指引。
原文摘要 · Abstract (English)
Generative AI has recently propelled the decoding of images from brain activity. How do these approaches scale with the amount and type of neural recordings? Here, we systematically compare image decoding from four types of non-invasive devices: electroencephalography (EEG), magnetoencephalography (MEG), high-field functional Magnetic Resonance Imaging (3T fMRI) and ultra-high field (7T) fMRI. For this, we evaluate decoding models on the largest benchmark to date, encompassing 8 public datasets, 84 volunteers, 498 hours of brain recording and 2.3 million brain responses to natural images. Unlike previous work, we focus on single-trial decoding performance to simulate real-time settings. This systematic comparison reveals three main findings. First, the most precise neuroimaging devices tend to yield the best decoding performances, when the size of the training sets are similar. However, the gain enabled by deep learning - in comparison to linear models - is obtained with the noisiest devices. Second, we do not observe any plateau of decoding performance as the amount of training data increases. Rather, decoding performance scales log-linearly with the amount of brain recording. Third, this scaling law primarily depends on the amount of data per subject. However, little decoding gain is observed by increasing the number of subjects. Overall, these findings delineate the path most suitable to scale the decoding of images from non-invasive brain recordings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。