提升视觉模型特征分辨率，让细节更清晰。
LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision Foundation Models
- 用坐标注意力机制融合高分辨图像与低分辨率特征
- 自蒸馏生成伪标签，训练效果显著优于现有方法
- 适配多种输入和特征分辨率，适合像素级任务
视觉基础模型（如DINOv2和CLIP）在众多下游任务中表现优异，但其特征分辨率有限，影响像素级理解应用。特征上采样为解决此问题提供了可行路径。本文识别出两个关键因素：上采样器架构与训练目标。针对架构，提出基于坐标的交叉注意力变换器，将高分辨率图像与坐标信息及低分辨率视觉基础模型特征融合，生成高质量、清晰的高分辨率特征。针对训练目标，提出利用类无关掩码和自蒸馏构建高分辨率伪真值特征。实验表明，该方法在多个下游任务中显著优于现有特征上采样技术，能有效捕捉细粒度细节，并灵活适应不同输入与特征分辨率。代码已开源。
原文摘要 · Abstract (English)
Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved impressive results on various downstream tasks, but their limited feature resolution hampers performance in applications requiring pixel-level understanding. Feature upsampling offers a promising direction to address this challenge. In this work, we identify two critical factors for enhancing feature upsampling: the upsampler architecture and the training objective. For the upsampler architecture, we introduce a coordinate-based cross-attention transformer that integrates the high-resolution images with coordinates and low-resolution VFM features to generate sharp, high-quality features. For the training objective, we propose constructing high-resolution pseudo-groundtruth features by leveraging class-agnostic masks and self-distillation. Our approach effectively captures fine-grained details and adapts flexibly to various input and feature resolutions. Through experiments, we demonstrate that our approach significantly outperforms existing feature upsampling techniques across various downstream tasks. Our code is released at https://github.com/andrehuang/loftup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。