用脑网络启发的专家路由模型,实现可解释且泛化的脑影像视觉重建。
MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding

- 基于脑区功能分组设计分层专家架构,模拟大脑网络分工。
- 跨被试重建精度高,仅需微调路由模块即可适配新个体。
- 路由机制透明揭示各脑区对图像语义与空间特征的影响,适合神经科学研究。
从fMRI解码视觉体验为理解人类感知和构建先进脑机接口提供了强大途径。然而,现有方法常以最大化重建保真度为目标,忽视了可解释性这一获取神经科学洞见的关键。为此,我们提出MoRE-Brain,一种兼具高保真、可适应与可解释性的神经启发框架。该框架采用分层混合专家架构,不同专家处理功能相关的体素组信号,模拟特定脑网络。专家先将fMRI编码至冻结的CLIP空间,再通过微调的扩散模型生成图像,由创新的双阶段路由机制动态加权专家贡献。主要贡献包括:1)基于脑网络原理设计的混合专家架构;2)通过共享核心专家网络、仅微调个体化路由模块,实现高效跨被试泛化;3)显式路由机制揭示各脑区如何影响重建图像的语义与空间属性,增强机制可解释性。大量实验验证其高重建保真度,瓶颈分析表明有效利用了真实神经信号,而非依赖生成先验。代码即将开源:https://github.com/yuxiangwei0808/MoRE-Brain。
原文摘要 · Abstract (English)
Decoding visual experiences from fMRI offers a powerful avenue to understand human perception and develop advanced brain-computer interfaces. However, current progress often prioritizes maximizing reconstruction fidelity while overlooking interpretability, an essential aspect for deriving neuroscientific insight. To address this gap, we propose MoRE-Brain, a neuro-inspired framework designed for high-fidelity, adaptable, and interpretable visual reconstruction. MoRE-Brain uniquely employs a hierarchical Mixture-of-Experts architecture where distinct experts process fMRI signals from functionally related voxel groups, mimicking specialized brain networks. The experts are first trained to encode fMRI into the frozen CLIP space. A finetuned diffusion model then synthesizes images, guided by expert outputs through a novel dual-stage routing mechanism that dynamically weighs expert contributions across the diffusion process. MoRE-Brain offers three main advancements: First, it introduces a novel Mixture-of-Experts architecture grounded in brain network principles for neuro-decoding. Second, it achieves efficient cross-subject generalization by sharing core expert networks while adapting only subject-specific routers. Third, it provides enhanced mechanistic insight, as the explicit routing reveals precisely how different modeled brain regions shape the semantic and spatial attributes of the reconstructed image. Extensive experiments validate MoRE-Brain's high reconstruction fidelity, with bottleneck analyses further demonstrating its effective utilization of fMRI signals, distinguishing genuine neural decoding from over-reliance on generative priors. Consequently, MoRE-Brain marks a substantial advance towards more generalizable and interpretable fMRI-based visual decoding. Code will be publicly available soon: https://github.com/yuxiangwei0808/MoRE-Brain.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。