SAILViT通过渐进特征优化，提升多模态大模型的视觉主干鲁棒性与泛化能力。
SAILViT: Towards Robust and Generalizable Visual Backbones for MLLMs via Gradual Feature Refinement
- 采用渐进式特征精炼实现粗到细的特征对齐
- 在多个数据集和模型规模上均显著提升性能
- 适合追求多模态交互稳定性的研究者使用
视觉变换器（ViTs）是构建多模态大语言模型（MLLMs）视觉理解能力的基础主干。尽管大多数ViTs通过图像-文本对的对比学习或自监督机制取得了优异表现，但在与大语言模型（LLMs）进行基于连接器的联合训练时，仍面临参数初始化冲突和模态语义差距的问题。为此，本文提出SAILViT，一种通过渐进特征精炼增强的视觉变换器，以促进MLLMs突破复杂多模态交互中的性能瓶颈。SAILViT通过逐步特征对齐与世界知识注入，更契合目标训练需求。我们进行了全面的实证分析，验证了SAILViT在不同参数规模、模型结构、训练策略和数据量下的强大鲁棒性与泛化能力。配备SAILViT后，现有MLLMs在OpenCompass基准上多个下游任务中均表现出显著且一致的性能提升。SAILViT系列模型已发布于https://huggingface.co/BytedanceDouyinContent。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) are essential as foundation backbones in establishing the visual comprehension capabilities of Multimodal Large Language Models (MLLMs). Although most ViTs achieve impressive performance through image-text pair-based contrastive learning or self-supervised mechanisms, they struggle to engage in connector-based co-training directly with LLMs due to potential parameter initialization conflicts and modality semantic gaps. To address the above challenges, this paper proposes SAILViT, a gradual feature learning-enhanced ViT for facilitating MLLMs to break through performance bottlenecks in complex multimodal interactions. SAILViT achieves coarse-to-fine-grained feature alignment and world knowledge infusion with gradual feature refinement, which better serves target training demands. We perform thorough empirical analyses to confirm the powerful robustness and generalizability of SAILViT across different dimensions, including parameter sizes, model architectures, training strategies, and data scales. Equipped with SAILViT, existing MLLMs show significant and consistent performance improvements on the OpenCompass benchmark across extensive downstream tasks. SAILViT series models are released at https://huggingface.co/BytedanceDouyinContent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。