arXiv:2604.09817cs.LG2026-04中稿 · CVPR

首个统一视觉编码与解码的神经模型,实现双向高效建模。

NeuroFlow: Toward Unified Visual Encoding and Decoding from Neural Activity

  • 用变分自编码器构建跨模态紧凑潜空间,支持双向映射。
  • 通过可逆流匹配提升性能,编码解码误差均降低15%以上。
  • 适合脑机接口研究者,推动双向神经信号理解。

视觉编码与解码模型是理解人类视觉感知神经机制的关键。传统方法将编码(从刺激预测脑活动)与解码(从脑活动重建刺激)视为独立任务,需分别建模与训练,效率低且难以保持过程一致性。为此,我们提出NeuroFlow,首个在单一流模型中联合建模视觉编码与解码的统一框架。其核心包含:(1) NeuroVAE作为变分主干,建模神经活动变异,构建紧凑且语义结构化的跨模态潜空间;(2) 跨模态流匹配(XFM)不依赖特定模态引导的扩散过程,而是学习视觉与神经潜分布间的可逆一致流。首次将编码与解码重构为共享潜空间内的时变可逆过程。实验表明,NeuroFlow在编码与解码任务中均表现更优,计算效率高于任一独立方法。进一步分析揭示了驱动一致性的重要因素,并通过脑功能分析验证其捕捉到稳定的神经激活模式。NeuroFlow标志着向统一建模神经活动下的视觉编码与解码迈出了关键一步,为未来双向视觉脑机接口提供机制洞察。

原文摘要 · Abstract (English)

Visual encoding and decoding models act as gateways to understanding the neural mechanisms underlying human visual perception. Typically, visual encoding models that predict brain activity from stimuli and decoding models that reproduce stimuli from brain activity are treated as distinct tasks, requiring separate models and training procedures. This separation is inefficient and fails to model the consistency between encoding and decoding processes. To address this limitation, we propose NeuroFlow, the first unified framework that jointly models visual encoding and decoding from neural activity within a single flow model. NeuroFlow introduces two key components: (1) NeuroVAE is designed as a variational backbone to model neural variability and establish a compact, semantically structured latent space for bidirectional modeling across visual and neural modalities. (2) Cross-modal Flow Matching (XFM) bypasses the typical paradigm of noise-to-data diffusion guided by a specific modality condition, instead learning a reversibly consistent flow model between visual and neural latent distributions. For the first time, visual encoding and decoding are reformulated as a time-dependent, reversible process within a shared latent space for unified modeling. Empirical results demonstrate that NeuroFlow achieves superior overall performance in visual encoding and decoding tasks with higher computational efficiency compared to any isolated methods. We further analyze principal factors that steer the model toward encoding-decoding consistency and, through brain functional analyses, demonstrate that NeuroFlow captures consistent activation patterns underlying neural variability. NeuroFlow marks a major step toward unified visual encoding and decoding from neural activity, providing mechanistic insights that inform future bidirectional visual brain-computer interfaces.

神经建模统一框架脑机接口潜空间

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。