让低分辨率视觉模型的特征图更清晰，提升细节感知能力。
FeatSharp: Your Vision Model Features, Sharper
- 通过低成本方法统一上采样低分辨率视觉特征图。
- 在语义分割等任务中显著提升细节识别效果。
- 适合需要高精度特征的感知与多模态模型训练场景。
视觉编码器的特征图是众多现代AI任务的基础，涵盖语义分割、目标检测、深度感知等核心感知算法，以及视觉-语言模型中的多模态理解。当前计算机视觉领域通用视觉主干网络的前沿是视觉变换器（ViT），通常使用对比损失（如CLIP）进行训练。然而，大多数现成的ViT模型，尤其是CLIP，存在分辨率固定且偏低的问题：多数运行在224×224像素，即使所谓的“高分辨率”版本也仅在378–448像素左右，且缺乏灵活性。本文提出一种新颖方法，能够以一致且低成本的方式上采样低分辨率视觉编码器的特征图，同时捕获原本因分辨率不足而丢失的细粒度细节。我们在核心感知任务及基于RADIO的聚合式模型训练中验证了该方法的有效性，证明其可为蒸馏提供更丰富的目标信号。代码已公开于https://github.com/NVlabs/FeatSharp。
原文摘要 · Abstract (English)
The feature maps of vision encoders are fundamental to myriad modern AI tasks, ranging from core perception algorithms (e.g. semantic segmentation, object detection, depth perception, etc.) to modern multimodal understanding in vision-language models (VLMs). Currently, in computer vision, the frontier of general purpose vision backbones is Vision Transformers (ViT), typically trained using contrastive loss (e.g. CLIP). A key problem with most off-the-shelf ViTs, particularly CLIP, is that these models are inflexibly low resolution. Most run at $224 \times 224$px, while the "high-resolution" versions are around $378-448$px, but still inflexible. We introduce a novel method to coherently and cheaply upsample the feature maps of low-resolution vision encoders while picking up on fine-grained details that would otherwise be lost due to resolution. We demonstrate the effectiveness of this approach on core perception tasks as well as within agglomerative model training using RADIO as a way of providing richer targets for distillation. Code available at https://github.com/NVlabs/FeatSharp .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。