用生成式视觉模型提升多模态大模型，效果优于主流方法。
Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion
- 采用生成式视觉编码器与深度广度融合架构，增强视觉表征
- 在多个基准上超越现有最佳模型，尤其在图文对齐与视觉理解任务中表现突出
- 适合研究多模态生成、视觉语言对齐及大模型微调的学者使用
我们提出 Florence-VL，一种基于 Florence-2 生成式视觉基础模型的新一代多模态大语言模型（MLLM）。与广泛使用的对比学习训练的 CLIP 式视觉变压器不同，Florence-2 能捕捉多层次、多维度的视觉特征，更具泛化性。我们设计了新颖的特征融合架构和训练方案，将 Florence-2 的视觉特征高效整合至 Phi 3.5 与 LLama 3 等预训练语言模型中。特别地，提出“深度-广度融合”（DBFusion）机制，融合不同深度层级及多提示下的视觉特征。模型训练采用端到端预训练，再对投影层与语言模型进行微调，基于多样化的开源数据集（包含高质量图像描述与指令对）。定量分析与可视化表明，该模型在视觉-语言对齐方面显著优于主流视觉编码器，其丰富的深度与广度特征起关键作用。Florence-VL 在涵盖通用 VQA、感知、幻觉抑制、OCR、图表理解、知识密集型理解等多任务基准上均实现显著提升。为促进后续研究，模型与完整训练流程已开源。
原文摘要 · Abstract (English)
We present Florence-VL, a new family of multimodal large language models (MLLMs) with enriched visual representations produced by Florence-2, a generative vision foundation model. Unlike the widely used CLIP-style vision transformer trained by contrastive learning, Florence-2 can capture different levels and aspects of visual features, which are more versatile to be adapted to diverse downstream tasks. We propose a novel feature-fusion architecture and an innovative training recipe that effectively integrates Florence-2's visual features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted from different depths and under multiple prompts. Our model training is composed of end-to-end pretraining of the whole model followed by finetuning of the projection layer and the LLM, on a carefully designed recipe of diverse open-source datasets that include high-quality image captions and instruction-tuning pairs. Our quantitative analysis and visualization of Florence-VL's visual features show its advantages over popular vision encoders on vision-language alignment, where the enriched depth and breath play important roles. Florence-VL achieves significant improvements over existing state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks covering general VQA, perception, hallucination, OCR, Chart, knowledge-intensive understanding, etc. To facilitate future research, our models and the complete training recipe are open-sourced. https://github.com/JiuhaiChen/Florence-VL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。