arXiv:2606.14024cs.CV2026-06

用ViT自身特征实现精准图像上采样,解决传统方法模糊与失真问题。

ViT-Up: Faithful Feature Upsampling for Vision Transformers

论文配图:ViT-Up: Faithful Feature Upsampling for Vision Transformers
图 1 · 摘自论文原文
  • 基于ViT中间层隐藏状态构建查询,无需外部图像引导
  • 在Cityscapes上提升最高2.07 mIoU,SPair-71k上提升8.09 [email protected]
  • 适配大模型,可扩展性强,适合密集视觉任务

视觉变换器(ViTs)已成为视觉表征学习的主流架构,提供强大且通用的主干特征。然而,由于全局自注意力的二次计算开销,ViTs通常在较小的块令牌网格上运行,这为语义分割、深度估计等密集预测任务带来了持续瓶颈。现有无任务依赖的上采样方法虽能生成视觉清晰的稠密表示,但依赖浅层图像编码器进行引导,易引入特征泄漏、碎片化和模糊。本文提出ViT-Up,一种隐式特征上采样框架,以中间ViT隐藏状态分层构造查询,替代外部图像引导。该方法可在任意连续图像坐标上预测特征,同时保持与主干特征空间对齐。实验表明,ViT-Up在密集预测和语义对应任务中持续优于当前最优图像引导上采样方法。在DINOv3-S+上,相较之前方法,Cityscapes上提升最高2.07 mIoU,SPair-71k上提升最高4.17 [email protected];使用更大规模的DINOv3-B主干时,性能进一步提升至+3.36 mIoU和+8.09 [email protected],证明了其随主干容量增长的良好可扩展性。

原文摘要 · Abstract (English)

Vision Transformers (ViTs) have become a dominant architecture for visual representation learning, providing exceptionally strong and broadly reusable backbone features. However, ViTs are commonly operated on relatively small patch-token grids due to the quadratic cost of global self-attention, which creates a persistent bottleneck for dense prediction tasks such as semantic segmentation and depth estimation. This has motivated the development of task-agnostic feature upsamplers. While recent state-of-the-art methods produce visually sharp dense representations, their reliance on shallow image encoders for guided upsampling can introduce feature leakage, fragmentation, and blur. We introduce ViT-Up, an implicit feature upsampling framework that replaces external image guidance with layer-wise query construction from intermediate ViT hidden states. This enables feature prediction at arbitrary continuous image coordinates while preserving alignment with the backbone feature space. Experiments demonstrate that ViT-Up consistently outperforms state-of-the-art image-guided upsamplers across dense prediction and semantic correspondence. On DINOv3-S+, ViT-Up improves over prior methods by up to +2.07 mIoU on Cityscapes and +4.17 [email protected] on SPair-71k. With the larger DINOv3-B backbone, these gains increase to +3.36 mIoU and +8.09 [email protected], demonstrating that ViT-Up scales favorably with backbone capacity.

ViT特征上采样语义分割稠密预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。