用单一解码器统一处理图文，提升效率和理解能力。
Optimizing Vision-Language Interactions Through Decoder-Only Models
- 采用自适应输入融合机制，无需独立视觉编码器
- 在4500万图文对上训练，多项任务超越现有模型
- 适合追求高效多模态推理的开发者与研究者
视觉语言模型（VLMs）已成为多模态任务的关键技术，但其依赖独立视觉编码器带来了效率低、可扩展性差和模态对齐困难等问题。为此，本文提出MUDAIF（Multimodal Unified Decoder with Adaptive Input Fusion），一种基于解码器的统一多模态模型，通过新型视觉标记适配器（VTA）和自适应交叉注意力机制，无缝融合视觉与文本输入。该模型摒弃了传统视觉编码器，实现了更高的效率、灵活性和跨模态理解能力。在包含4500万图像-文本对的大规模数据集上训练后，MUDAIF在VQA、图像描述生成和多模态推理等多个基准测试中持续优于当前最优方法。大量分析与人工评估表明，MUDAIF具备强鲁棒性、良好泛化能力及实际可用性，确立了无编码器视觉语言模型的新标准。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have emerged as key enablers for multimodal tasks, but their reliance on separate visual encoders introduces challenges in efficiency, scalability, and modality alignment. To address these limitations, we propose MUDAIF (Multimodal Unified Decoder with Adaptive Input Fusion), a decoder-only vision-language model that seamlessly integrates visual and textual inputs through a novel Vision-Token Adapter (VTA) and adaptive co-attention mechanism. By eliminating the need for a visual encoder, MUDAIF achieves enhanced efficiency, flexibility, and cross-modal understanding. Trained on a large-scale dataset of 45M image-text pairs, MUDAIF consistently outperforms state-of-the-art methods across multiple benchmarks, including VQA, image captioning, and multimodal reasoning tasks. Extensive analyses and human evaluations demonstrate MUDAIF's robustness, generalization capabilities, and practical usability, establishing it as a new standard in encoder-free vision-language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。