让大模型聚焦关键区域，用更少的计算资源看清细节。
Zoomer: Adaptive Image Focus Optimization for Black-box MLLM
- 动态分配注意力，重点增强重要视觉区域。
- 在9个基准上提升准确率最高达27%，图像令牌减少67%。
- 适合无法访问内部结构的商用多模态模型使用。
多模态大语言模型（如GPT-4o、Gemini Pro、Claude 3.5）虽能统一处理文本与视觉输入，但在真实场景中常因小物体或精细空间信息丢失而产生幻觉。我们指出两大根源：缺乏区域自适应注意力和固定令牌预算导致均匀下采样。为此提出Zoomer——一种面向黑盒MLLM的视觉提示框架，通过（1）提示感知强调模块突出语义相关区域，（2）保持空间关系的编排策略，（3）根据预算动态分配全局与局部细节的令牌。在九个基准和三个商用模型上的实验表明，Zoomer最高提升准确率27%，同时将图像令牌使用量降低67%。该方法为不可见模型内部的鲁棒、资源感知多模态理解提供了系统性解决方案。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) such as GPT-4o, Gemini Pro, and Claude 3.5 have enabled unified reasoning over text and visual inputs, yet they often hallucinate in real world scenarios especially when small objects or fine spatial context are involved. We pinpoint two core causes of this failure: the absence of region-adaptive attention and inflexible token budgets that force uniform downsampling, leading to critical information loss. To overcome these limitations, we introduce Zoomer, a visual prompting framework that delivers token-efficient, detail-preserving image representations for black-box MLLMs. Zoomer integrates (1) a prompt-aware emphasis module to highlight semantically relevant regions, (2) a spatial-preserving orchestration schema to maintain object relationships, and (3) a budget-aware strategy to adaptively allocate tokens between global context and local details. Extensive experiments on nine benchmarks and three commercial MLLMs demonstrate that Zoomer boosts accuracy by up to 27% while cutting image token usage by up to 67%. Our approach establishes a principled methodology for robust, resource-aware multimodal understanding in settings where model internals are inaccessible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。