用专家网络提升多模态3D理解，性能显著超越现有方法。
MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding
- 引入专家混合机制，让不同专家专注处理特定模态或跨模态交互。
- 在Multi3DRefer上比顶尖模型高6.1 mIoU，多任务表现优异。
- 适合研究多模态3D感知、视觉语言理解的开发者参考。
多模态3D理解是计算机视觉的基础任务。以往的多模态融合方法通常采用单一密集融合网络，难以应对模态间显著的异质性和复杂性，导致性能不佳。本文提出MoE3D，将混合专家（MoE）机制融入多模态学习框架。核心思想是部署一组专用“专家”网络，各自擅长处理特定模态或跨模态交互模式。具体地，设计基于MoE的Transformer以更好地利用视觉特征中的互补信息；提出信息聚合模块进一步提升融合效果；采用Top-1门控机制，使每组专家仅处理对应特征，保障高效性。此外，设计渐进式预训练策略，更好利用语义和2D先验知识，获得良好初始化。MoE3D在四个主流3D理解任务中表现卓越，尤其在Multi3DRefer上比当前最优方法高出6.1 mIoU。
原文摘要 · Abstract (English)
Multi-modal 3D understanding is a fundamental task in computer vision. Previous multi-modal fusion methods typically employ a single, dense fusion network, struggling to handle the significant heterogeneity and complexity across modalities, leading to suboptimal performance. In this paper, we propose MoE3D, which integrates Mixture of Experts (MoE) into the multi-modal learning framework. The core is that we deploy a set of specialized "expert" networks, each adept at processing a specific modality or a mode of cross-modal interaction. Specifically, the MoE-based transformer is designed to better utilize the complementary information hidden in the visual features. Information aggregation module is put forward to further enhance the fusion performance. Top-1 gating is employed to make one expert process features with expert groups, ensuring high efficiency. We further propose a progressive pre-training strategy to better leverage the semantic and 2D prior, thus equipping the network with good initialization. Our MoE3D achieves competitive performance across four prevalent 3D understanding tasks. Notably, our MoE3D surpasses the top-performing counterpart by 6.1 mIoU on Multi3DRefer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。