arXiv:2512.15372cs.IRcs.AI2025-12被引 1

让视觉模型根据图像复杂度自动调节计算量,提速20%且不丢精度。

Image Complexity-Aware Adaptive Retrieval for Efficient Vision-Language Models

  • 按图像复杂度动态决定计算深度,简单图早退,复杂图全走。
  • 在保持类别级性能不变的前提下,编码速度提升20%,实例级性能保留95%。
  • 无需重排就能直接匹配图文,适合需要高效推理的部署场景。

视觉语言模型中的视觉变换器通常对每张图像使用相同的计算量,无论其复杂程度如何。本文提出ICAR(图像复杂度感知自适应检索),一种自适应计算方法,使视觉变换器对简单图像使用较少计算量,而对复杂图像则通过完整网络深度处理。核心挑战在于保持跨模态对齐:不同处理深度产生的嵌入必须与文本匹配兼容。ICAR通过双路径训练,确保早期退出和完整深度路径生成的嵌入在语义空间中兼容。与需昂贵重排序的现有两阶段方法不同,ICAR支持直接图文匹配,无额外开销。为确定计算量,我们设计了ConvNeXt-IC,将图像复杂度评估视为分类任务。采用现代分类器主干而非专用架构,在人类标注上达到0.959的皮尔逊相关系数,同时实现4.4倍加速。在包含真实网络数据的标准基准测试中,ICAR实现20%的图像编码加速,保持类别级性能,并达到95%的实例级性能,推动视觉语言系统可持续扩展。

原文摘要 · Abstract (English)

Vision transformers in vision-language models typically use the same amount of compute for every image, regardless of whether it is simple or complex. We propose ICAR (Image Complexity-Aware Retrieval), an adaptive computation approach that enables vision transformers to use less compute for simple images whilst processing complex images through their full network depth. The key challenge is maintaining cross-modal alignment: embeddings from different processing depths must remain compatible for text matching. ICAR solves this through dual-path training that produces compatible embeddings from both the early-exit and full-depth paths. This maintains compatibility between image representations and text embeddings in the same semantic space, whether an image exits early or processes fully. Unlike existing two-stage approaches that require expensive reranking, ICAR enables direct image-text matching without additional overhead. To determine how much compute to use, we develop ConvNeXt-IC, which treats image complexity assessment as a classification task. By applying modern classifier backbones rather than specialised architectures, ConvNeXt-IC achieves state-of-the-art performance, attaining a Pearson correlation coefficient of 0.959 with human labelling whilst delivering 4.4x faster complexity prediction. Evaluated on standard benchmarks augmented with real-world web data, ICAR achieves 20% faster image encoding while maintaining category-level performance and 95% of instance-level performance, enabling sustainable scaling of vision-language systems.

自适应计算视觉语言效率优化多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。