用低密度脑电设备实现清晰视觉重建,提升真实场景脑机应用可行性
EEG2Vision: A Multimodal EEG-Based Framework for 2D Visual Reconstruction in Cognitive Neuroscience
- 基于脑电信号的端到端图像重建框架,融合扩散模型与语义引导增强
- 24通道下仍保持可辨视觉结构,感知质量提升9.71%(IS指标)
- 适合追求实时脑-图像交互的科研或临床应用,尤其关注低硬件成本场景
从非侵入式脑电图(EEG)重建视觉刺激仍面临空间分辨率低、噪声大等挑战,尤其在现实中的低密度电极配置下。为此,我们提出EEG2Vision,一种模块化、端到端的脑电到图像重建框架,系统评估不同脑电分辨率(128、64、32、24通道)下的重建性能,并通过提示引导的后处理增强机制提升视觉质量。该方法以脑电条件扩散重建为基础,利用多模态大语言模型提取语义描述,结合图像到图像扩散模型优化几何结构与感知一致性,同时保留脑电驱动的结构特征。实验表明,随着通道数减少,语义解码准确率显著下降(如50分类Top-1准确率从89%降至38%),而重建质量轻微下降(如FID从76.77升至80.51)。所提增强机制在所有配置下均持续提升感知指标,在低通道设置中最高获得9.71%的IS增益。用户研究证实,增强后的重建结果在感知上更受青睐。该方法显著提升了使用低分辨率脑电设备进行实时脑-图像应用的可行性,有望推动此类技术走出实验室。
原文摘要 · Abstract (English)
Reconstructing visual stimuli from non-invasive electroencephalography (EEG) remains challenging due to its low spatial resolution and high noise, particularly under realistic low-density electrode configurations. To address this, we present EEG2Vision, a modular, end-to-end EEG-to-image framework that systematically evaluates reconstruction performance across different EEG resolutions (128, 64, 32, and 24 channels) and enhances visual quality through a prompt-guided post-reconstruction boosting mechanism. Starting from EEG-conditioned diffusion reconstruction, the boosting stage uses a multimodal large language model to extract semantic descriptions and leverages image-to-image diffusion to refine geometry and perceptual coherence while preserving EEG-grounded structure. Our experiments show that semantic decoding accuracy degrades significantly with channel reduction (e.g., 50-way Top-1 Acc from 89% to 38%), while reconstruction quality slight decreases (e.g., FID from 76.77 to 80.51). The proposed boosting consistently improves perceptual metrics across all configurations, achieving up to 9.71% IS gains in low-channel settings. A user study confirms the clear perceptual preference for boosted reconstructions. The proposed approach significantly boosts the feasibility of real-time brain-2-image applications using low-resolution EEG devices, potentially unlocking this type of applications outside laboratory settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。