arXiv:2512.02351cs.CVcs.AI2025-12被引 4

发现统一多模态模型中生成模块可压缩性差,提出稀疏激活方法提升效率。

Understanding and Harnessing Sparsity in Unified Multimodal Models

  • 用无训练剪枝分析模型各组件,发现生成部分对压缩敏感
  • 通过专家混合机制实现稀疏激活,仅用一半参数达全模型性能
  • 适合追求高效推理的多模态模型部署者

大型多模态模型在理解和生成任务上取得显著进展。近期工作致力于构建统一框架,将异构组件整合以支持双重能力,但这种统一带来推理效率低下问题:特定任务或样本未必需要完整模型知识与容量。然而,对这些低效性如何在不同组件中体现仍缺乏系统理解。本文首次采用无训练剪枝作为探测手段,系统分析统一多模态模型组件,涵盖深度剪枝和宽度缩减。研究发现,理解组件在理解和生成任务中均表现出显著可压缩性,且生成任务中更明显;而生成组件对压缩极为敏感,即使在中等压缩比下性能也急剧下降。为此,我们提出受样本间动态激活模式启发的专家混合(MoE)适配方法:将生成模块划分为多个专家,实现稀疏激活以恢复生成质量。通过专家冻结微调验证稀疏激活有效性,并进一步证明全可训练适配带来额外增益。最终,适配后的BAGEL模型仅激活约一半参数,性能却接近完整模型。代码已开源。

原文摘要 · Abstract (English)

Large multimodal models have achieved remarkable progress in both understanding and generation. Recent efforts pursue unified multimodal models that integrate heterogeneous components to support both capabilities within a single framework. However, such unification introduces inference inefficiencies, e.g., specific tasks or samples may not require the full knowledge or capacity of the unified model. Yet, a systematic understanding of how these inefficiencies manifest across different components remains limited. In this work, we first conduct a systematic analysis of unified multimodal model components using training-free pruning as a probing methodology, considering both depth pruning and width reduction. Our study reveals that the understanding component exhibits notable compressibility in both understanding and generation tasks, which is more pronounced in the latter. In contrast, the generation components are highly sensitive to compression, with performance deteriorating sharply even under moderate compression ratios. To address this limitation, we propose the Mixture-of-Experts (MoE) Adaptation, inspired by the dynamic activation patterns observed across different samples. This approach partitions the generation module into multiple experts and enables sparse activation to restore generation quality. We validate the effectiveness of sparse activation through expert-frozen tuning and further demonstrate that a fully trainable adaptation delivers additional gains. As a result, the adapted BAGEL model achieves performance comparable to the full model while activating only about half of its parameters. The code is released at \href{https://github.com/Shwai-He/SparseUnifiedModel}{this link}.

多模态稀疏性MoE高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。