自适应视觉粒度让多模态模型更高效，少算快跑不丢精度。
AVG-LLaVA: An Efficient Large Multimodal Model with Adaptive Visual Granularity
- 根据图像和指令动态选择视觉细节层级，减少冗余计算。
- 在AI2D上视觉令牌减少85.3%，推理速度提升2.53倍。
- 无需人工标注，通过新训练范式让模型自动学会选粒度。
近期大型多模态模型（LMMs）取得了显著进展。处理高分辨率图像时，主流方法通常将图像划分为多个局部区域和一个全局图像，导致大量视觉令牌。本文提出AVG-LLaVA，一种能根据输入图像和指令自适应选择视觉粒度的LMM。首先通过多层池化获取不同粒度的视觉令牌；然后设计视觉粒度路由模块，包含Transformer、MLP和投票层，基于图像与指令选择最优粒度。此外，提出RGLF训练范式，使路由预测的粒度与模型偏好对齐，无需额外人工标注数据。大量实验表明，AVG-LLaVA在11个基准测试中表现优异，显著降低视觉令牌数量并加速推理（如在AI2D上视觉令牌减少85.3%，推理速度提升2.53×）。
原文摘要 · Abstract (English)
Recently, large multimodal models (LMMs) have achieved significant advancements. When dealing with high-resolution images, dominant LMMs typically divide them into multiple local images and a global image, leading to a large number of visual tokens. In this work, we introduce AVG-LLaVA, an LMM that can adaptively select the appropriate visual granularity based on the input image and instruction. Specifically, we first apply the multiple pooling layers to obtain visual tokens at different granularities. Then we propose a visual granularity router, which includes a Transformer layer, an MLP layer, and a voter layer, used to select the appropriate visual granularity based on the image and instruction. Furthermore, we put forward RGLF, a novel training paradigm that aims at aligning the granularity predicted by the router with the preferences of the LMM, without the need for additional manually annotated data. Extensive experiments and analysis show that AVG-LLaVA achieves superior performance across 11 benchmarks, as well as significantly reduces the number of visual tokens and speeds up inference (e.g., an 85.3% reduction in visual tokens and a 2.53$\times$ increase in inference speed on the AI2D benchmark).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。