通过文本引导选择视觉标记,大幅减少计算量同时保持精度。
FlashVLM: Text-Guided Visual Token Selection for Large Multimodal Models
- 用文本与图像的跨模态相似度动态筛选关键视觉标记
- 在94.4%压缩率下仍保持92.8%准确率,优于未压缩基线
- 适合需要高效推理的大规模多模态模型部署
大型视觉语言模型(VLMs)通常对每张图像或视频帧处理数百至数千个视觉标记,带来二次方注意力开销和显著冗余。现有标记缩减方法常忽略文本查询或依赖深层注意力图,其在激进剪枝下的不稳定性导致语义对齐下降。本文提出FlashVLM,一种文本引导的视觉标记选择框架,能动态适应查询。不依赖噪声注意力权重,而是计算投影图像标记与语言模型空间中归一化文本嵌入间的显式跨模态相似度。该外部相关性与内部视觉显著性结合,采用对数域加权与温度控制锐化。此外,多样性保留分区确保最小但具代表性的背景标记被保留,以维持全局上下文。在相同标记预算与评估协议下,FlashVLM实现无损压缩超越,即使在LLaVA 1.5上剪枝达77.8%视觉标记时仍略超未剪枝基线,并在94.4%压缩率下保持92.8%准确率。14个图像与视频基准上的大量实验表明,FlashVLM在效率-性能权衡上达到当前最优,且在主流VLM中展现出强鲁棒性与泛化能力。
原文摘要 · Abstract (English)
Large vision-language models (VLMs) typically process hundreds or thousands of visual tokens per image or video frame, incurring quadratic attention cost and substantial redundancy. Existing token reduction methods often ignore the textual query or rely on deep attention maps, whose instability under aggressive pruning leads to degraded semantic alignment. We propose FlashVLM, a text guided visual token selection framework that dynamically adapts visual inputs to the query. Instead of relying on noisy attention weights, FlashVLM computes an explicit cross modal similarity between projected image tokens and normalized text embeddings in the language model space. This extrinsic relevance is fused with intrinsic visual saliency using log domain weighting and temperature controlled sharpening. In addition, a diversity preserving partition retains a minimal yet representative set of background tokens to maintain global context. Under identical token budgets and evaluation protocols, FlashVLM achieves beyond lossless compression, slightly surpassing the unpruned baseline while pruning up to 77.8 percent of visual tokens on LLaVA 1.5, and maintaining 92.8 percent accuracy even under 94.4 percent compression. Extensive experiments on 14 image and video benchmarks demonstrate that FlashVLM delivers state of the art efficiency performance trade offs while maintaining strong robustness and generalization across mainstream VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。