让ViT高效处理任意分辨率图像，提升细粒度分割精度
SPAR: Single-Pass Any-Resolution ViT for Open-vocabulary Segmentation
- 用单次遍历替代滑窗，实现任意分辨率输入的快速推理
- 在开放词汇分割任务中，最高提升10.5 mIoU，超越教师模型
- 无需修改架构或像素级标注，适合高分辨率密集预测场景
基础视觉变压器（ViTs）在需要精细空间理解的任务中表现有限，原因在于其固定预训练分辨率和固有的粗粒度块级表示。这一问题在密集预测任务（如基于ViT的视觉语言模型进行开放词汇分割）中尤为突出，高分辨率输入对像素级推理至关重要。现有方法通常在预训练分辨率下采用滑窗策略处理大分辨率图像，虽通过更细步长提升准确率，但计算开销巨大。我们提出SPAR：一种针对高效高分辨率推理设计的无分辨率依赖密集特征提取器。通过特征回归损失，将细步长滑窗教师模型的空间推理能力蒸馏到单次遍历的学生模型中，无需架构改动或像素级监督。应用于开放词汇分割任务，SPAR使单次遍历基线提升最高达10.5 mIoU，甚至超越教师模型，证明了其在高效高分辨率推理中的有效性。代码见：https://github.com/naomikombol/SPAR
原文摘要 · Abstract (English)
Foundational Vision Transformers (ViTs) have limited effectiveness in tasks requiring fine-grained spatial understanding, due to their fixed pre-training resolution and inherently coarse patch-level representations. These challenges are especially pronounced in dense prediction scenarios, such as open-vocabulary segmentation with ViT-based vision-language models, where high-resolution inputs are essential for accurate pixel-level reasoning. Existing approaches typically process large-resolution images using a sliding-window strategy at the pre-training resolution. While this improves accuracy through finer strides, it comes at a significant computational cost. We introduce SPAR: Single-Pass Any-Resolution ViT, a resolution-agnostic dense feature extractor designed for efficient high-resolution inference. We distill the spatial reasoning capabilities of a finely-strided, sliding-window teacher into a single-pass student using a feature regression loss, without requiring architectural changes or pixel-level supervision. Applied to open-vocabulary segmentation, SPAR improves single-pass baselines by up to 10.5 mIoU and even surpasses the teacher, demonstrating effectiveness in efficient, high-resolution reasoning. Code: https://github.com/naomikombol/SPAR
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。