无需训练即可提升视觉模型细粒度预测能力，解决像素级语义分割难题。
PixelUp: Zero-Shot Semantic Feature Upsampling for Fine-Grained Vision Tasks
- 通过多尺度语义引导的窗口交叉注意力链实现零样本特征上采样
- 在语义分割上提升1.2 mIoU，深度估计提升0.25 δ₁，跨模型通用
- 适用于无监督、开放词汇分割，无需额外训练，适合部署在各类视觉模型上
自监督视觉基础模型（VFMs）因其强大的可迁移表征已成为下游任务的核心。然而，其补丁-标记级别的特征对需要高精度细粒度预测的任务（如语义分割、深度估计）而言过于粗糙。现有特征上采样方法存在局限：可学习上采样器通常针对特定编码器设计，需重新训练；基于图像的引导方法常引入纹理伪影且缺乏语义指导。本文提出PixelUp，一种零样本、无需依赖具体视觉基础模型的上采样器，通过粗到细的窗口化交叉注意力架构，利用多尺度语义特征实现语义感知。实验表明，PixelUp在密集预测任务中优于所有现有方法，在不同VFMs上平均提升1.2 mIoU（语义分割）和0.25 δ₁（NYUv2深度估计）。此外，它在免训练的开放词汇和无监督语义分割任务中分别提升1.3 mIoU和0.5 mIoU。代码已开源。
原文摘要 · Abstract (English)
Self-supervised Vision Foundation Models (VFMs) have become essential backbones for downstream tasks due to their strong and transferable visual representations. However, their patch-token-level features are often too coarse for dense prediction tasks such as semantic segmentation and depth estimation when accurate fine-grained predictions are required. Feature upsampling methods have been developed to recover pixel-level detail but still face limitations. Learnable upsamplers are often designed for a specific encoders and must be retrained for different encoders. Image-guided methods that use shallow pixel encoders often introduce textural artifacts and lack the semantic guidance needed for accurate downstream predictions. We introduce PixelUp, a zero-shot VFM-agnostic upsampler achieving semantic awareness through a coarse-to-fine chain of windowed cross-attention architecture guided by multi-scale semantic features. We demonstrate that PixelUp outperforms both VFM-specific and VFM-agnostic upsamplers, achieving state-of-the-art performance on dense prediction tasks with an average improvement of +1.2 mIoU on semantic segmentation and +0.25 $δ_1$, on NYUv2 depth estimation across VFMs. PixelUp further improves training-free open-vocabulary and unsupervised semantic segmentation by an average of +1.3 mIoU and +0.5 mIoU, respectively. Code available at https://pixelup-project.vercel.app/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。