多尺度视觉重采样提升多模态模型精度，计算开销极小。
MS-Resampler: Multi-Scope Visual Resampling for Efficient Multimodal LLMs
- 通过引入空间范围先验，实现多粒度视觉信息聚合。
- 在10个公开数据集上均超越传统单尺度方法，性能更优。
- 适合追求高精度与低延迟的多模态应用开发人员。
多模态大语言模型通常采用基于重采样的投影器，将密集的视觉特征转化为紧凑的标记序列以供语言建模。现有方法大多依赖全局交叉注意力的单一固定聚合范围，容易模糊细粒度局部信息，且在固定标记预算下难以同时捕捉局部细节与全局上下文。本文提出MS-Resampler，一种用于多模态大语言模型的多范围视觉重采样框架。该框架通过在重采样注意力中注入显式的空间范围先验，构建多个特定范围的重采样分支，使每个分支能够从局部到全局不同粒度聚合视觉信息。这些范围特异性重采样器的输出随后被自适应融合，生成最终的视觉表示用于语言建模。在10个公共多模态基准上的大量实验表明，相较于传统单范围重采样器，MS-Resampler在保持极低计算开销的同时，持续提升了视觉理解与多模态推理能力。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) typically employ resampling-based projectors to transform dense visual features into a compact token sequence for language modeling. Most existing resamplers adopt a single, fixed aggregation scope via global cross-attention, which can blur fine-grained local evidence and limit the ability to capture both local details and global context within a fixed token budget. In this work, we propose MS-Resampler, a multi-scope visual resampling framework for MLLMs. MS-Resampler instantiates multiple scope-specific resamplers by injecting explicit spatial scope priors into the resampling attention, enabling each branch to aggregate visual information at a particular granularity from local to global. The outputs of these scope-specific resamplers are then adaptively fused to produce the final visual representations for language modeling. Extensive experiments on ten public multimodal benchmarks show that MS-Resampler consistently improves visual understanding and multimodal reasoning over conventional single-scope resamplers, while introducing only minimal computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。