用脑电波生成图像，实现低成本实时脑机接口
Guess What I Think: Streamlined EEG-to-Image Generation with Latent Diffusion Models
- 基于控制网的潜空间扩散模型，直接用脑电信号生成图像
- 在多个基准上优于现有方法，仅需简单预处理
- 适合追求高效实时脑机接口的研究者
从脑电波生成图像正受到越来越多关注，因其有望提升脑机接口系统对视觉信息编码的理解能力。以往研究多集中于高分辨率的fMRI-to-Image任务，但fMRI成本高昂且无法实现实时交互。相比之下，脑电图（EEG）具有低成本、非侵入性和便携性优势，更适合未来实时应用。然而，EEG存在空间分辨率低、易受噪声和伪影干扰等问题，导致图像生成难度大。本文提出一种基于控制网适配器的简化框架，通过脑电信号条件化潜空间扩散模型（LDM），有效解决上述挑战。我们在多个流行基准上进行了实验与消融研究，结果表明该方法超越了现有先进模型。与需要复杂预处理、预训练、多种损失函数和描述生成模型的方法不同，本方法简洁高效，仅需最少预处理和少量组件。代码已开源：https://github.com/LuigiSigillo/GWIT。
原文摘要 · Abstract (English)
Generating images from brain waves is gaining increasing attention due to its potential to advance brain-computer interface (BCI) systems by understanding how brain signals encode visual cues. Most of the literature has focused on fMRI-to-Image tasks as fMRI is characterized by high spatial resolution. However, fMRI is an expensive neuroimaging modality and does not allow for real-time BCI. On the other hand, electroencephalography (EEG) is a low-cost, non-invasive, and portable neuroimaging technique, making it an attractive option for future real-time applications. Nevertheless, EEG presents inherent challenges due to its low spatial resolution and susceptibility to noise and artifacts, which makes generating images from EEG more difficult. In this paper, we address these problems with a streamlined framework based on the ControlNet adapter for conditioning a latent diffusion model (LDM) through EEG signals. We conduct experiments and ablation studies on popular benchmarks to demonstrate that the proposed method beats other state-of-the-art models. Unlike these methods, which often require extensive preprocessing, pretraining, different losses, and captioning models, our approach is efficient and straightforward, requiring only minimal preprocessing and a few components. The code is available at https://github.com/LuigiSigillo/GWIT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。