arXiv:2512.18635cs.CV2025-12

用脑电波直接生成和编辑图像,实现跨模态视觉创作。

Uni-Neur2Img: Unified Neural Signal-Guided Image Generation, Editing, and Stylization via Diffusion Transformers

  • 通过可插拔的轻量级模块注入脑电信号,灵活支持多模态条件生成。
  • 在三个数据集上实现高质量图像生成、局部编辑与风格迁移,计算开销低。
  • 适合脑机接口、神经科学与创意生成领域的研究者与开发者。

从神经信号直接生成或编辑图像在神经科学、计算机视觉与脑机交互交叉领域具有巨大潜力。本文提出Uni-Neur2Img,一个统一的神经信号驱动图像生成与编辑框架。该框架引入基于LoRA的参数高效神经信号注入模块,独立处理每种条件信号,作为可插拔组件,实现无需修改基础模型参数的灵活多模态条件生成;同时采用因果注意力机制以满足长序列建模需求。现有神经驱动生成研究多聚焦于文本模态作为条件或中间表示,对视觉模态作为直接条件信号的探索有限。为填补这一空白,我们构建了EEG-Style数据集。我们在公开基准和自收集神经信号数据集上进行了全面评估:(1) 在公开的CVPR40数据集上进行脑电驱动图像生成;(2) 在公开的Loongx数据集上进行语义感知的局部图像编辑;(3) 在自收集的EEG-Style数据集上实现脑电驱动风格迁移。大量实验结果表明,该方法在生成保真度、编辑一致性与风格迁移质量方面均有显著提升,同时保持低计算开销并具备强扩展性。因此,Uni-Neur2Img提供了一种统一、高效且可扩展的神经信号与视觉内容生成桥梁解决方案。

原文摘要 · Abstract (English)

Generating or editing images directly from Neural signals has immense potential at the intersection of neuroscience, vision, and Brain-computer interaction. In this paper, We present Uni-Neur2Img, a unified framework for neural signal-driven image generation and editing. The framework introduces a parameter-efficient LoRA-based neural signal injection module that independently processes each conditioning signal as a pluggable component, facilitating flexible multi-modal conditioning without altering base model parameters. Additionally, we employ a causal attention mechanism accommodate the long-sequence modeling demands of conditional generation tasks. Existing neural-driven generation research predominantly focuses on textual modalities as conditions or intermediate representations, resulting in limited exploration of visual modalities as direct conditioning signals. To bridge this research gap, we introduce the EEG-Style dataset. We conduct comprehensive evaluations across public benchmarks and self-collected neural signal datasets: (1) EEG-driven image generation on the public CVPR40 dataset; (2) neural signal-guided image editing on the public Loongx dataset for semantic-aware local modifications; and (3) EEG-driven style transfer on our self-collected EEG-Style dataset. Extensive experimental results demonstrate significant improvements in generation fidelity, editing consistency, and style transfer quality while maintaining low computational overhead and strong scalability to additional modalities. Thus, Uni-Neur2Img offers a unified, efficient, and extensible solution for bridging neural signals and visual content generation.

脑机接口图像生成扩散模型神经信号

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。