让多模态大模型按需感知图像,提速近4.4倍且不丢精度。
Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Models
- 根据任务需求动态选择高分辨率处理区域,避免全图冗余计算。
- 在文档与高清场景上推理速度提升2.52至4.39倍,精度保持甚至超越基线。
- 适用于需要精细视觉理解的场景,如OCR、文档分析及新兴视觉思考模型。
多模态大模型(MLLMs)在文档理解与密集场景感知等细粒度任务中需高分辨率视觉输入。然而,现有全局分辨率提升方法盲目将冗余视觉令牌注入二次自注意力机制,严重拖慢推理速度,且忽略空间稀疏性与查询意图。为此,我们提出Q-Zoom:一种查询感知的自适应高分辨率感知框架,采用高效粗到精策略。首先,轻量级动态门控网络在粗粒度特征已足够时跳过高分辨率处理;其次,针对需细粒度感知的查询,自蒸馏区域提议网络(SD-RPN)直接从中间特征空间精准定位任务相关感兴趣区域(RoI)。为高效优化,门控网络采用一致性感知生成策略获取确定性路由标签,而SD-RPN则采用全自监督蒸馏范式。通过连续时空对齐与针对性微调,将密集局部RoI与粗粒度全局布局无缝融合。大量实验表明,Q-Zoom建立了显著的帕累托前沿。以Qwen2.5-VL-7B为基准,在文档与OCR基准上推理加速2.52倍,在高分辨率场景下加速4.39倍,同时保持基线峰值精度;当配置为最大感知保真度时,分别在两项基准上超越基线1.1%和8.1%。这些提升可无缝迁移至Qwen3-VL、LLaVA及新兴基于强化学习的视觉思维模型。
原文摘要 · Abstract (English)
MLLMs require high-resolution visual inputs for fine-grained tasks like document understanding and dense scene perception. However, current global resolution scaling paradigms indiscriminately flood the quadratic self-attention mechanism with visually redundant tokens, severely bottlenecking inference throughput while ignoring spatial sparsity and query intent. To overcome this, we propose Q-Zoom, a query-aware adaptive high-resolution perception framework that operates in an efficient coarse-to-fine manner. First, a lightweight Dynamic Gating Network safely bypasses high-resolution processing when coarse global features suffice. Second, for queries demanding fine-grained perception, a Self-Distilled Region Proposal Network (SD-RPN) precisely localizes the task-relevant Region-of-Interest (RoI) directly from intermediate feature spaces. To optimize these modules efficiently, the gating network uses a consistency-aware generation strategy to derive deterministic routing labels, while the SD-RPN employs a fully self-supervised distillation paradigm. A continuous spatio-temporal alignment scheme and targeted fine-tuning then seamlessly fuse the dense local RoI with the coarse global layout. Extensive experiments demonstrate that Q-Zoom establishes a dominant Pareto frontier. Using Qwen2.5-VL-7B as a primary testbed, Q-Zoom accelerates inference by 2.52 times on Document & OCR benchmarks and 4.39 times in High-Resolution scenarios while matching the baseline's peak accuracy. Furthermore, when configured for maximum perceptual fidelity, Q-Zoom surpasses the baseline's peak performance by 1.1% and 8.1% on these respective benchmarks. These robust improvements transfer seamlessly to Qwen3-VL, LLaVA, and emerging RL-based thinking-with-image models. Project page is available at https://yuhengsss.github.io/Q-Zoom/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。