让视觉大模型在推理时同时用多分辨率图像,提升识别效果。
MuRF: Unlocking the Multi-Scale Potential of Vision Foundation Models
- 用冻结的视觉大模型处理同一图像的多个分辨率,融合特征。
- 在多个任务上显著提升性能,如ImageNet分类准确率提升1.8%。
- 无需训练,适配DINOv2、SigLIP2等主流模型,通用性强。
视觉基础模型(VFMs)已成为现代计算机视觉的核心,能为多种任务提供鲁棒表征。尽管近期进展使这些模型可在训练中处理不同输入尺寸,但推理阶段通常仍局限于单一固定尺度。这种普遍的单尺度范式忽略了视觉感知的一个基本特性:不同分辨率提供互补的归纳偏置——低分辨率擅长全局语义识别,高分辨率则对细粒度特征提取至关重要。本文提出多分辨率融合(MuRF),一种简单但通用有效的推理时策略,以利用这一协同效应。MuRF不依赖单一视角,而是通过冻结的视觉基础模型在多个分辨率下处理图像,并融合所得特征,构建统一表征。其最大优势在于普适性:不绑定特定架构,可作为无需训练的视觉表征增强手段。我们通过实验证明,该方法适用于多种关键计算机视觉任务,覆盖多个不同的视觉基础模型家族,主要基于DINOv2,也成功扩展至对比学习模型如SigLIP2。
原文摘要 · Abstract (English)
Vision Foundation Models (VFMs) have become the cornerstone of modern computer vision, offering robust representations across a wide array of tasks. While recent advances allow these models to handle varying input sizes during training, inference typically remains restricted to a single, fixed scale. This prevalent single-scale paradigm overlooks a fundamental property of visual perception: varying resolutions offer complementary inductive biases, where low-resolution views excel at global semantic recognition and high-resolution views are essential for fine-grained refinement. In this work, we propose Multi-Resolution Fusion (MuRF), a simple yet universally effective strategy to harness this synergy at inference time. Instead of relying on a single view, MuRF constructs a unified representation by processing an image at multiple resolutions through a frozen VFM and fusing the resulting features. The universality of MuRF is its most compelling attribute. It is not tied to a specific architecture, serving instead as a fundamental, training-free enhancement to visual representation. We empirically validate this by applying MuRF to a broad spectrum of critical computer vision tasks across multiple distinct VFM families - primarily DINOv2, but also demonstrating successful generalization to contrastive models like SigLIP2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。