智能选择图像分辨率，节省80%算力还保持准确
CARES: Context-Aware Resolution Selector for VLMs
- 根据上下文自动预测完成任务所需的最低图像分辨率
- 在多个基准上实现最高80%的计算量降低，性能几乎不变
- 轻量级模块，适配各类视觉语言模型，适合部署优化
大型视觉语言模型（VLM）通常以原生或高分辨率处理图像，导致视觉标记占比高达97%-99%，带来高昂的计算开销和延迟。本文提出CARES——一种上下文感知的分辨率选择器，作为轻量级预处理模块，在给定图像-查询对时，预测完成任务所需的最小输入分辨率。CARES利用一个小型VLM（350M）提取特征，并判断目标预训练VLM的回答是否收敛到最优表现。尽管训练时为离散分类器，推理时可插值连续分辨率，实现精细控制。在涵盖文档与自然图像的五个多模态基准上，以及多种目标VLM中，CARES在保持任务性能的同时，最高可减少80%的计算量。
原文摘要 · Abstract (English)
Large vision-language models (VLMs) commonly process images at native or high resolution to remain effective across tasks. This inflates visual tokens ofter to 97-99% of total tokens, resulting in high compute and latency, even when low-resolution images would suffice. We introduce \emph{CARES}-a \textbf{C}ontext-\textbf{A}ware \textbf{R}esolution \textbf{S}elector, a lightweight preprocessing module that, given an image-query pair, predicts the \emph{minimal} sufficient input resolution. CARES uses a compact VLM (350M) to extract features and predict when a target pretrained VLM's response converges to its peak ability to answer correctly. Though trained as a discrete classifier over a set of optional resolutions, CARES interpolates continuous resolutions at inference for fine-grained control. Across five multimodal benchmarks spanning documents and natural images, as well as diverse target VLMs, CARES preserves task performance while reducing compute by up to 80%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。