根据图像复杂度动态调整分辨率,提升视觉Mamba效率
MambaScope: Coarse-to-Fine Scoping for Efficient Vision Mamba
- 先粗后细:大块处理图像,复杂区域再精细重算
- 低置信度时仅重算关键区域,额外计算极少
- 适合追求高效推理的视觉Mamba应用
视觉Mamba虽是视觉Transformer的高效替代方案,但其效率受限于输入标记数量。现有标记缩减方法多通过剪枝或合并降低计算量,但会损失信息。尤其当所有图像均采用相同细粒度处理时问题更严重。我们观察到:简单图像无需精细处理,复杂图像才需细化。基于此提出MambaScope,一种自适应高效推理框架。该框架首先将图像划分为大块进行粗粒度推理,大幅减少标记数和计算量;当模型预测置信度较低时,仅对选定区域以更高分辨率重处理,以最小代价恢复细节。这种动态分辨率分配策略使计算资源随图像复杂度自适应分配,在不牺牲精度的前提下显著提升效率。跨多种视觉任务的实验表明,MambaScope在准确率与效率上均优于基线Vision Mamba及当前最优的标记缩减技术。
原文摘要 · Abstract (English)
Vision Mamba has emerged as a promising and efficient alternative to Vision Transformers, yet its efficiency remains fundamentally constrained by the number of input tokens. Existing token reduction approaches typically adopt token pruning or merging to reduce computation. However, they inherently lead to information loss as they discard or compress token representations. This problem is further exacerbated when the same fine-grained token processing is uniformly applied across all images regardless of visual complexity. We observe that not all inputs require fine-grained processing: simple images can be effectively handled at a coarse resolution, while only complex ones require refinement. Based on this insight, we propose MambaScope, an adaptive framework for efficient inference for Vision Mamba. MambaScope first performs coarse-grained inference by dividing the input image into large patches, significantly reducing token length and computation. When the model's prediction confidence is low, selected regions are re-processed at a finer resolution to recover essential visual details with minimal additional cost. This dynamic resolution assignment strategy allows MambaScope to allocate computation adaptively according to image complexity, achieving efficient processing without compromising accuracy. Experiments across various vision tasks demonstrate that MambaScope outperforms both the baseline Vision Mamba and state-of-the-art token reduction techniques in terms of accuracy and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。