提出渐进对齐的多专家视觉模型,解决专家负载不均问题。
Astrea: A MOE-based Visual Understanding Model with Progressive Alignment
- 用四类专用专家构建视觉理解矩阵,覆盖检测、分割等任务。
- 通过渐进预对齐与概率残差连接,提升专家间知识协同效率。
- 适合需要多任务泛化能力的通用多模态系统研发者。
基于混合专家(MoE)架构的视觉语言模型已成为多模态理解的核心范式,但任务多样性导致异构视觉专家间的负载失衡,优化单一专家常损害其他专家性能。为此,我们提出Astrea,一种基于渐进预对齐的多专家协作视觉语言模型。其核心创新包括:1)融合检测、分割、分类、描述四类专用模型的异构专家协调机制,构建涵盖关键视觉理解要素的专家矩阵;2)动态知识融合策略,通过对比学习在模型隐空间实现专家渐进预对齐,并结合概率激活的随机残差连接以保持知识连续性;3)增强优化框架,采用动量对比学习建模长程依赖,及自适应权重分配器实时校准专家贡献。在涵盖视觉问答、图像描述与跨模态检索的12个基准任务上评估显示,Astrea相比现有最优模型平均性能提升+4.7%。本研究首次实证证明渐进预对齐能克服任务异构性限制,为通用多模态智能体开发奠定新方法基础。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) based on Mixture-of-Experts (MoE) architectures have emerged as a pivotal paradigm in multimodal understanding, offering a powerful framework for integrating visual and linguistic information. However, the increasing complexity and diversity of tasks present significant challenges in coordinating load balancing across heterogeneous visual experts, where optimizing one specialist's performance often compromises others' capabilities. To address task heterogeneity and expert load imbalance, we propose Astrea, a novel multi-expert collaborative VLM architecture based on progressive pre-alignment. Astrea introduces three key innovations: 1) A heterogeneous expert coordination mechanism that integrates four specialized models (detection, segmentation, classification, captioning) into a comprehensive expert matrix covering essential visual comprehension elements; 2) A dynamic knowledge fusion strategy featuring progressive pre-alignment to harmonize experts within the VLM latent space through contrastive learning, complemented by probabilistically activated stochastic residual connections to preserve knowledge continuity; 3) An enhanced optimization framework utilizing momentum contrastive learning for long-range dependency modeling and adaptive weight allocators for real-time expert contribution calibration. Extensive evaluations across 12 benchmark tasks spanning VQA, image captioning, and cross-modal retrieval demonstrate Astrea's superiority over state-of-the-art models, achieving an average performance gain of +4.7\%. This study provides the first empirical demonstration that progressive pre-alignment strategies enable VLMs to overcome task heterogeneity limitations, establishing new methodological foundations for developing general-purpose multimodal agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。