arXiv:2607.24424cs.CV2026-07

多尺度自适应视觉编码器减少视觉令牌数,提升细粒度感知与推理效率

MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning

  • 融合浅层、中间和深层特征,通过位置门控保留局部结构与全局语义
  • 根据问题动态路由令牌,复杂图像自动增加令牌预算,平均令牌数降为146个
  • 适用于需要高效多模态推理的场景,如低延迟视觉问答系统

视觉语言模型通常将预训练视觉编码器生成的所有视觉令牌输入大语言模型。然而,最终层特征会丢失文本、局部属性和空间关系,高分辨率输入则显著增加上下文长度与推理延迟。本文提出 method,一种多尺度自适应视觉编码器。该方法利用位置依赖门控融合视觉Transformer的浅层、中间和深层特征,在保留全局语义的同时增强边缘、文本和局部结构。随后根据问题相关性、局部信息量、全局语义和空间覆盖范围进行条件化令牌路由,并依据图像复杂度自适应调整令牌预算。为缓解压缩损失,引入全量到压缩表示的蒸馏机制与空间多样性正则项。在统一7B语言模型框架下的模拟实验中, method将SigLIP-SO400M视觉令牌平均数量从729降至146(约80.0%),在VQAv2、GQA、TextVQA、ScienceQA-IMG和MMBench上平均得分提升2.2个百分点,单图首个输出时间由228毫秒降至129毫秒。所有报告数据目前仅为论文组织与实验设计占位符;正式结论需经真实训练、独立复现与官方评测验证。

原文摘要 · Abstract (English)

Vision-language models commonly project all tokens produced by a pretrained vision encoder into a large language model. However, final-layer features can discard text, local attributes, and spatial relationships, while high-resolution inputs substantially increase context length and inference latency. We introduce \method, a Multi-scale Adaptive Vision Encoder. \method uses position-dependent gates to fuse shallow, intermediate, and deep features from a vision Transformer, preserving global semantics while enhancing edges, text, and local structure. It then performs question-conditioned token routing according to question relevance, local information content, global semantics, and spatial coverage, with a token budget that adapts to image complexity. To mitigate compression loss, we further introduce full-to-compressed representation distillation and a spatial diversity regularizer. In an illustrative simulation under a unified 7B language-model framework, \method reduces the average number of SigLIP-SO400M visual tokens from 729 to 146 (approximately 80.0\%) and improves the mean score on VQAv2, GQA, TextVQA, ScienceQA-IMG, and MMBench by 2.2 percentage points, while reducing single-image time to first token from 228\,ms to 129\,ms. We provide the full model design and evaluation protocol. All reported numbers currently serve only as placeholders for paper organization and experimental design; formal claims require real training runs, independent replications, and official benchmark evaluation.

视觉编码器多模态推理令牌压缩视觉感知

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。