让视觉语言模型自动决定每张图用多少视觉标记,更省计算资源。
AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition
- 先用低分辨率图粗略处理,必要时调用框选工具精准补采关键区域。
- 在多个VQA数据集上用更少视觉标记达到更高准确率,比现有方法省30%以上。
- 适合追求高效推理的视觉问答系统开发者,尤其关注资源受限场景。
视觉语言模型在视觉问答任务中表现优异,但依赖大量视觉标记带来显著计算开销。现有高效方法采用固定比例压缩视觉标记,被动运行且无法适应不同任务需求。为此,我们提出AdaptVision,一种基于粗到精策略的自适应视觉标记获取范式。模型初始使用低分辨率图像的压缩视觉标记,必要时调用边界框工具裁剪关键区域以选择性补充信息。通过强化学习框架训练,平衡准确率与效率。核心是解耦回合策略优化(DTPO),将学习目标分为:(1) 工具使用优化,提升工具调用正确性;(2) 准确率提升,改进回答质量。进一步对优势估计进行解耦,分别计算各目标相关标记的优势。相比标准GRPO,该方法优化更有效。在多个VQA基准测试中,AdaptVision实现更优性能,同时显著减少视觉标记消耗,优于当前最先进高效模型。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。