用轻量结构提升视觉语言模型效率与细粒度理解能力
Firebolt-VL: Efficient Vision-Language Understanding with Cross-Modality Modulation
- 用液态基础模型替代Transformer解码器，实现线性推理时间
- 通过词元-网格相关模块，精准定位图文匹配的视觉区域
- 适合资源受限场景，如智能摄像头、文档理解等实际应用
近年来多模态大模型在视觉语言理解上取得显著进展，但其高计算成本限制了在个人助手、文档理解及智能摄像头等资源受限场景中的部署。现有方法多依赖具有二次复杂度的Transformer交叉注意力机制，制约效率。小型视觉语言模型往往难以精确捕捉细粒度任务相关的视觉区域，影响细粒度推理性能。为此，我们提出Firebolt-VL，采用液态基础模型（LFM）解码器替代Transformer解码器，并引入词元-网格相关模块，通过状态空间模型结合FiLM条件调节，计算文本词元与图像块间的轻量级相关性，实现对提示相关视觉区域的动态强调。该设计支持线性时间推理。在多个基准测试中，Firebolt-VL在保持高效的同时实现了准确的细粒度理解。模型与代码已公开于：https://fireboltvl.github.io
原文摘要 · Abstract (English)
Recent advances in multimodal large language models (MLLMs) have enabled impressive progress in vision-language understanding, yet their high computational cost limits deployment in resource-constrained scenarios such as personal assistants, document understanding, and smart cameras. Most existing methods rely on Transformer-based cross-attention, whose quadratic complexity hinders efficiency. Moreover, small vision-language models often struggle to precisely capture fine-grained, task-relevant visual regions, leading to degraded performance on fine-grained reasoning tasks that limit their effectiveness in the real world. To address these issues, we introduce Firebolt-VL, an efficient vision-language model that replaces the Transformer-based decoder with a Liquid Foundation Model (LFM) decoder. To further enhance visual grounding, we propose a Token-Grid Correlation Module, which computes lightweight correlations between text tokens and image patches and modulates via the state-space model with FiLM conditioning. This enables the model to selectively emphasize visual regions relevant to the textual prompt while maintaining linear-time inference. Experimental results across multiple benchmarks demonstrate that Firebolt-VL achieves accurate, fine-grained understanding with significantly improved efficiency. Our model and code are available at: https://fireboltvl.github.io
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。