通过限制视觉信息量,强制视觉语言模型主动感知。
Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth

- 用视觉带宽约束迫使模型主动搜索关键区域。
- 在多个基准上平均提升5%的准确率。
- 无需修改架构或额外损失,可直接接入现有训练流程。
将视觉语言模型(VLMs)部署于高分辨率视觉环境中的情境化代理需要主动感知能力——即通过缩放、裁剪和平移等操作动态决定观察位置。然而,当前训练范式生成的模型仅模仿此类操作的表象,而未真正依赖其输出,这种现象被称为懒惰感知。我们发现其根源在于学习不对称性:当粗粒度全局视图结合语言先验即可达到中等精度时,模型缺乏学习多步视觉搜索的动力。若模型无需主动观察即可成功,则永远不会学会主动看。为此,我们提出「饿以感知」(Starve to Perceive)训练范式,通过限制每次观测的视觉信息量(紧缩令牌预算),使单次视角无法完成任务,从而迫使模型必须进行主动感知。该方法无需辅助损失、奖励设计或架构修改,作为标准后训练流程的最小插件式改进,模型在多样基准上实现平均5%的相对性能提升。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) deployed as situated agents in high-resolution visual environments require active perception -- the ability to dynamically decide where to look through operations like zooming, cropping, and panning. However, current training paradigms produce models that mimic the surface form of such operations without functionally depending on their outputs, a phenomenon we term lazy perception. We trace this to a fundamental learning asymmetry: when coarse global views combined with language priors suffice for moderate accuracy, the model has no incentive to learn harder multi-step visual search. If a model can succeed without actively looking, it will never learn to look. This motivates Starve to Perceive, a training paradigm that constrains visual bandwidth -- restricting each observation to a tight token budget so that no single view suffices for task completion, making active perception the only viable strategy. Despite requiring no auxiliary losses, reward shaping, or architectural changes -- serving as a minimal, plug-in modification to standard post-training pipelines -- models trained under perceptual starvation achieve substantial gains of 5% average relative improvement across diverse benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。