针对视觉语言模型中模态差异，提出新型专家架构提升多模态理解能力。
AsyMoE: Leveraging Modal Asymmetry for Enhanced Expert Specialization in Large Vision-Language Models
- 设计三类专家：模态内、跨模态层级交互、强调证据的语言专家
- 在多个基准上准确率比普通MoE高26.58%，激活参数少25.45%
- 适合追求高效多模态建模的开发者和研究者
大型视觉语言模型（LVLMs）通过规模化架构和大规模训练，在多模态任务上表现优异。然而，现有混合专家（MoE）方法面临视觉与语言处理间的不对称性挑战：视觉信息空间完整，而语言需保持序列上下文。分析发现，深层语言专家逐渐失去上下文依赖，更依赖参数化知识而非输入的视觉与语言信息。为此，我们提出AsyMoE，通过三类专用专家建模该不对称性：用于模态特异性处理的内部专家、用于层次化跨模态交互的双曲专家，以及抑制参数偏倚、保持上下文连贯性的证据优先语言专家。大量实验表明，AsyMoE相比基线MoE提升26.58%准确率，较模态特定MoE提升15.45%，且激活参数量比密集模型减少25.45%。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have demonstrated impressive performance on multimodal tasks through scaled architectures and extensive training. However, existing Mixture of Experts (MoE) approaches face challenges due to the asymmetry between visual and linguistic processing. Visual information is spatially complete, while language requires maintaining sequential context. As a result, MoE models struggle to balance modality-specific features and cross-modal interactions. Through systematic analysis, we observe that language experts in deeper layers progressively lose contextual grounding and rely more on parametric knowledge rather than utilizing the provided visual and linguistic information. To address this, we propose AsyMoE, a novel architecture that models this asymmetry using three specialized expert groups. We design intra-modality experts for modality-specific processing, hyperbolic inter-modality experts for hierarchical cross-modal interactions, and evidence-priority language experts to suppress parametric biases and maintain contextual grounding. Extensive experiments demonstrate that AsyMoE achieves 26.58% and 15.45% accuracy improvements over vanilla MoE and modality-specific MoE respectively, with 25.45% fewer activated parameters than dense models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。