用最优传输对齐脑影像,提升解码精度并揭示神经信息冗余与协同。
Optimal Transport for Brain-Image Alignment: Unveiling Redundancy and Synergy in Neural Information Processing
- 基于最优传输构建脑区与图像嵌入间的全局匹配机制
- 在十项指标上超越当前最佳方法,单人训练提升6.11%
- 可揭示脑区信息冗余与协同,适用于脑信号解码研究
人工神经网络的设计受人类大脑结构启发,同时也能用于解读和理解脑信号。现有方法主要使用均方误差(MSE)对齐脑信号与刺激信号,仅关注局部点对点匹配,忽略全局对应关系,导致解释粗糙且解码不准确。本文通过最优传输(OT)解决这些问题,并从理论上证明了OT比MSE更有效的原理。具体而言,我们构建脑像素嵌入与图像嵌入之间的传输计划,实现更精确的匹配;通过控制传输量,缓解冗余信息的影响。我们将该对齐模型直接应用于脑图像描述任务,将脑信号输入大语言模型(LLM)而非图像。实验显示,在十项评估指标上达到当前最优性能,单人训练平均提升6.11%,跨人训练提升3.81%。此外,通过区域遮蔽与数据维度降低可视化实验,我们揭示了脑信息处理中的冗余性与协同性,结论与已有脑科学研究一致。我们认为该方法为未来更精准理解脑信号开辟了新路径。代码已公开于https://github.com/NKUShaw/OT-Alignment4brain-to-image。
原文摘要 · Abstract (English)
The design of artificial neural networks (ANNs) is inspired by the structure of the human brain, and in turn, ANNs offer a potential means to interpret and understand brain signals. Existing methods primarily align brain signals with stimulus signals using Mean Squared Error (MSE), which focuses only on local point-wise alignment and ignores global matching, leading to coarse interpretations and inaccuracies in brain signal decoding. In this paper, we address these issues through optimal transport (OT) and theoretically demonstrate why OT provides a more effective alignment strategy than MSE. Specifically, we construct a transport plan between brain voxel embeddings and image embeddings, enabling more precise matching. By controlling the amount of transport, we mitigate the influence of redundant information. We apply our alignment model directly to the Brain Captioning task by feeding brain signals into a large language model (LLM) instead of images. Our approach achieves state-of-the-art performance across ten evaluation metrics, surpassing the previous best method by an average of 6.11\% in single-subject training and 3.81\% in cross-subject training. Additionally, we have uncovered several insightful conclusions that align with existing brain research. We unveil the redundancy and synergy of brain information processing through region masking and data dimensionality reduction visualization experiments. We believe our approach paves the way for a more precise understanding of brain signals in the future. The code is available at https://github.com/NKUShaw/OT-Alignment4brain-to-image.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。