FLARE通过深度视觉语言融合,显著提升跨模态理解能力。
FLARE: Fully Integration of Vision-Language Representations for Deep Cross-Modal Understanding
- 视觉编码时融入文本信息,实现像素级对齐
- 解码阶段根据上下文聚合特征,达成查询级整合
- 仅用630个视觉标记即超越更大模型,适合高效应用
我们提出FLARE,一种具有全视角-语言对齐与融合范式的视觉语言模型家族。不同于依赖单一MLP投影器进行模态对齐并推迟跨模态交互至LLM解码的现有方法,FLARE在全流程中实现深度动态整合。核心贡献包括:(1) 文本引导的视觉编码,在视觉编码阶段引入文本信息以实现像素级对齐;(2) 上下文感知的对齐解码,在解码阶段基于文本上下文聚合视觉特征,实现查询级整合;(3) 双语义映射损失,从双模态监督特征映射,实现模态间桥梁构建;(4) 文本驱动的VQA合成,利用高质量文本生成对应图文对并合成图像,实现数据层面优化。我们在固定与动态分辨率设置下分别训练了3B和8B规模的FLARE模型,结果表明其全模态对齐显著优于现有方法且保持强泛化能力。FLARE 3B仅使用630个视觉标记即超越Cambrian-1 8B和Florence-VL 8B。消融实验显示,其性能优越且计算开销极低。即使无动态分辨率,仍优于LLaVA-NeXT,验证了方法有效性。代码、模型权重及数据集已开源于https://github.com/starriver030515/FLARE。
原文摘要 · Abstract (English)
We introduce FLARE, a family of vision language models (VLMs) with a fully vision-language alignment and integration paradigm. Unlike existing approaches that rely on single MLP projectors for modality alignment and defer cross-modal interaction to LLM decoding, FLARE achieves deep, dynamic integration throughout the pipeline. Our key contributions include: (1) Text-Guided Vision Encoding that incorporates textual information during vision encoding to achieve pixel-level alignment; (2) Context-Aware Alignment Decoding that aggregates visual features conditioned on textual context during decoding for query-level integration; (3) Dual-Semantic Mapping Loss to supervise feature mapping from both modalities and enable modality-level bridging; and (4) Text-Driven VQA Synthesis that leverages high-quality text to generate VQA pairs and synthesize corresponding images, enabling data-level optimization. We train FLARE at 3B and 8B scales under both fixed and dynamic resolution settings, demonstrating that our full-modality alignment significantly outperforms existing methods while maintaining strong generalizability. FLARE 3B surpasses Cambrian-1 8B and Florence-VL 8B using only 630 vision tokens. Ablation studies reveal that FLARE achieves superior performance over existing methods with minimal computational cost. Even without dynamic resolution, FLARE outperforms LLaVA-NeXT, validating the effectiveness of our approach. We release our code, model weights, and dataset in https://github.com/starriver030515/FLARE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。