用交互分割评测视觉大模型的特征上采样方法
Benchmarking Feature Upsampling Methods for Vision Foundation Models using Interactive Segmentation
- 以交互分割为基准，评估不同上采样策略对视觉大模型特征的提升效果
- 合适的上采样方法显著改善低分辨率特征质量，提升分割精度
- 适合关注视觉大模型细节理解与特征重构的研究者
视觉基础模型（Vision Foundation Models, VFMs）是大规模预训练模型，可作为多种计算机视觉任务的通用主干网络。随着其广泛应用，人们对它们在密集预测任务中的表现日益关注。然而，VFMs通常生成低分辨率特征，限制了其在该类任务中的直接应用。一种解决方案是采用与任务无关的特征上采样模块，以提升特征分辨率。为评估该方法的有效性，我们提出将交互分割（Interactive Segmentation, IS）作为新基准，用于评估VFMs上的特征上采样方法。由于其包含图像和用户点击的多模态输入，以及密集掩码输出，IS构建了一个需要全面视觉理解的挑战性环境。实验表明，选择合适的上采样策略能显著提升VFM特征质量。代码已开源：https://github.com/havrylovv/iSegProbe。
原文摘要 · Abstract (English)
Vision Foundation Models (VFMs) are large-scale, pre-trained models that serve as general-purpose backbones for various computer vision tasks. As VFMs' popularity grows, there is an increasing interest in understanding their effectiveness for dense prediction tasks. However, VFMs typically produce low-resolution features, limiting their direct applicability in this context. One way to tackle this limitation is by employing a task-agnostic feature upsampling module that refines VFM features resolution. To assess the effectiveness of this approach, we investigate Interactive Segmentation (IS) as a novel benchmark for evaluating feature upsampling methods on VFMs. Due to its inherent multimodal input, consisting of an image and a set of user-defined clicks, as well as its dense mask output, IS creates a challenging environment that demands comprehensive visual scene understanding. Our benchmarking experiments show that selecting appropriate upsampling strategies significantly improves VFM features quality. The code is released at https://github.com/havrylovv/iSegProbe
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。