用稀疏专家结构让多模态大模型稳定扩展，解决任务冲突问题。
Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of Experts
- 采用稀疏激活的专家混合架构，分担不同任务的处理
- 在多个基准上实现多任务性能提升，避免互相干扰
- 基于低秩适配（LoRA）设计，训练推理更快
随着多模态大语言模型（MLLM）研究日益深入，先进模型需同时处理多种文本与视觉任务（如VQA、检测、OCR和ChartQA），以满足真实应用场景。然而，由于不同任务数据在表征和分布上的显著差异，简单混合所有任务数据会引发著名的“多任务冲突”问题，导致各任务性能下降。为解决此问题，我们提出Awaker2.5-VL，一种适用于MLLM的专家混合（MoE）架构，通过多个稀疏激活的专家实现多任务能力。为加速训练与推理，模型中每个专家均采用低秩适配（LoRA）结构。在多个最新基准上的大量实验验证了该方法的有效性。代码与模型权重已在项目页面公开：https://github.com/MetabrainAGI/Awaker。
原文摘要 · Abstract (English)
As the research of Multimodal Large Language Models (MLLMs) becomes popular, an advancing MLLM model is typically required to handle various textual and visual tasks (e.g., VQA, Detection, OCR, and ChartQA) simultaneously for real-world applications. However, due to the significant differences in representation and distribution among data from various tasks, simply mixing data of all tasks together leads to the well-known``multi-task conflict" issue, resulting in performance degradation across various tasks. To address this issue, we propose Awaker2.5-VL, a Mixture of Experts~(MoE) architecture suitable for MLLM, which acquires the multi-task capabilities through multiple sparsely activated experts. To speed up the training and inference of Awaker2.5-VL, each expert in our model is devised as a low-rank adaptation (LoRA) structure. Extensive experiments on multiple latest benchmarks demonstrate the effectiveness of Awaker2.5-VL. The code and model weight are released in our Project Page: https://github.com/MetabrainAGI/Awaker.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。