用脑电等神经信号实现无需双手的图像编辑,让残障人士也能自由创作。
Neural-Driven Image Editing
- 融合脑电、近红外等多模态神经信号,通过动态融合模块提取用户意图。
- 在23,928对图像编辑数据上训练,性能接近文本驱动方法。
- 适合残障用户或无法使用传统输入方式的人群,推动无障碍创意技术发展。
传统图像编辑依赖手动操作,对运动或语言能力受限者不友好。本文提出LoongX,一种基于多模态神经生理信号的无手图像编辑方法。该方法利用23,928对图像编辑样本及其同步采集的脑电(EEG)、功能近红外(fNIRS)、光电容积脉搏波(PPG)和头部运动信号,训练先进的扩散模型。通过跨尺度状态空间(CS3)模块提取各模态特征,并以动态门控融合(DGF)模块整合为统一潜在表示,再通过微调扩散变换器(DiT)对齐编辑语义。预训练中采用对比学习将认知状态与自然语言意图对齐。大量实验表明,仅用神经信号时,性能接近文本驱动方法(CLIP-I: 0.6605 vs. 0.6558;DINO: 0.4812 vs. 0.4636),结合语音时更优(CLIP-T: 0.2588 vs. 0.2549)。结果证明神经驱动生成模型在可及性与直观性上的巨大潜力,为认知驱动的创意技术开辟新方向。代码与数据集已开源。
原文摘要 · Abstract (English)
Traditional image editing typically relies on manual prompting, making it labor-intensive and inaccessible to individuals with limited motor control or language abilities. Leveraging recent advances in brain-computer interfaces (BCIs) and generative models, we propose LoongX, a hands-free image editing approach driven by multimodal neurophysiological signals. LoongX utilizes state-of-the-art diffusion models trained on a comprehensive dataset of 23,928 image editing pairs, each paired with synchronized electroencephalography (EEG), functional near-infrared spectroscopy (fNIRS), photoplethysmography (PPG), and head motion signals that capture user intent. To effectively address the heterogeneity of these signals, LoongX integrates two key modules. The cross-scale state space (CS3) module encodes informative modality-specific features. The dynamic gated fusion (DGF) module further aggregates these features into a unified latent space, which is then aligned with edit semantics via fine-tuning on a diffusion transformer (DiT). Additionally, we pre-train the encoders using contrastive learning to align cognitive states with semantic intentions from embedded natural language. Extensive experiments demonstrate that LoongX achieves performance comparable to text-driven methods (CLIP-I: 0.6605 vs. 0.6558; DINO: 0.4812 vs. 0.4636) and outperforms them when neural signals are combined with speech (CLIP-T: 0.2588 vs. 0.2549). These results highlight the promise of neural-driven generative models in enabling accessible, intuitive image editing and open new directions for cognitive-driven creative technologies. The code and dataset are released on the project website: https://loongx1.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。