arXiv:2605.29591cs.AI2026-05

用离散扩散统一七种脑-视-语任务,让大脑信号能跨模态理解生成。

Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion

论文配图:Mind-Omni: A Unified Multi-Task Framework for Brain-Vision-Language Modeling via Discrete Diffusion
图 1 · 摘自论文原文
  • 通过离散扩散将脑电信号转为统一标记,实现多模态直接交互。
  • 在七项任务上达到新基准,性能媲美甚至超过专用大模型。
  • 适合脑机接口、神经建模与通用神经表征研究者参考。

脑-机接口领域中,外部刺激与内部神经表征的相互作用是核心研究方向。以往工作受限于单一任务专用模型,缺乏灵活性且忽略任务间协同效应。为此,我们提出Mind-Omni,首个通过离散扩散范式统一七类编码与解码任务的通用框架。其核心是一个新型脑信号分词器,可将异构连续脑信号转换为标准化离散标记,实现在共享语义空间中任意两模或多模态间的直接、逐标记交互与生成。为进一步激发高级推理能力,我们构建了专门的脑问答(BQA)指令微调数据集。该模型不仅在多任务统一框架中创下新纪录,还提供了多任务协同的有力证据:性能可比甚至超越更大规模的专用模型,为神经建模提供全新范式,并推动神经活动基础模型的发展。代码已开源:https://github.com/ReedOnePeck/Mind-Omni。

原文摘要 · Abstract (English)

Modeling the interplay between external stimuli and internal neural representations is a pivotal research area for Brain-Computer Interfaces (BCIs). A major limitation of prior work is the prevailing paradigm of specialized, single-task models, which curtails versatility and neglects inter-task synergies. To address this, we propose Mind-Omni, the first versatile framework that unifies seven distinct encoding and decoding tasks through a discrete diffusion paradigm. At its core is a novel Brain Tokenizer that transforms heterogeneous, continuous brain signals into standardized, discrete tokens. This enables direct, token-level interactions for mutual understanding and generation between any two or more modalities within a shared semantic space. To unlock advanced reasoning capabilities, we further curate a specialized Brain Question Answering (BQA) instruction-tuning dataset. Our model not only establishes a new state-of-the-art among multi-task unified frameworks but also provides strong evidence for multi-task synergy. By demonstrating performance competitive with, and at times superior to, larger specialized models, our work offers a powerful new paradigm for neural modeling and paves the way for foundation models of neural activity. The code is publicly available at https://github.com/ReedOnePeck/Mind-Omni.

脑机接口多模态建模扩散模型神经表征

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。