arXiv:2507.13348cs.CVcs.AI2025-07NeurIPS被引 39

通过强化学习动态调整图像分辨率,让视觉语言模型更智能高效。

VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning

论文配图:VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning
图 1 · 摘自论文原文
  • 用强化学习决定是否需要高分辨率图像,按需处理
  • 在文字识别任务上准确率提升,普通任务节省75%视觉令牌
  • 适合追求效率与精度平衡的视觉语言模型应用

近期视觉语言模型(VLMs)通过增加视觉令牌数量提升了性能,但这些令牌往往远超文本令牌。我们发现多数真实场景并不需要如此多的视觉令牌;尽管在部分OCR任务中性能下降明显,但在大多数通用VQA任务中,仅使用1/4分辨率仍可保持准确。为此,我们提出一种新范式——VisionThink,从低分辨率图像开始,智能判断是否足够解决问题;否则输出特殊令牌请求更高分辨率。相比固定压缩比的现有方法,VisionThink能针对不同样本自主决策。实验表明,该方法在OCR任务上具备更强细粒度理解能力,同时在简单任务中显著减少视觉令牌。我们采用强化学习,并设计了基于大模型评分的评判策略,结合精心设计的奖励函数与惩罚机制,实现稳定合理的图像重采样调用比例。大量实验证明了方法的优越性、高效性与有效性。代码已开源。

原文摘要 · Abstract (English)

Recent advancements in vision-language models (VLMs) have improved performance by increasing the number of visual tokens, which are often significantly longer than text tokens. However, we observe that most real-world scenarios do not require such an extensive number of visual tokens. While the performance drops significantly in a small subset of OCR-related tasks, models still perform accurately in most other general VQA tasks with only 1/4 resolution. Therefore, we propose to dynamically process distinct samples with different resolutions, and present a new paradigm for visual token compression, namely, VisionThink. It starts with a downsampled image and smartly decides whether it is sufficient for problem solving. Otherwise, the model could output a special token to request the higher-resolution image. Compared to existing Efficient VLM methods that compress tokens using fixed pruning ratios or thresholds, VisionThink autonomously decides whether to compress tokens case by case. As a result, it demonstrates strong fine-grained visual understanding capability on OCR-related tasks, and meanwhile saves substantial visual tokens on simpler tasks. We adopt reinforcement learning and propose the LLM-as-Judge strategy to successfully apply RL to general VQA tasks. Moreover, we carefully design a reward function and penalty mechanism to achieve a stable and reasonable image resize call ratio. Extensive experiments demonstrate the superiority, efficiency, and effectiveness of our method. Our code is available at https://github.com/dvlab-research/VisionThink.

视觉语言模型强化学习图像压缩高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。