用多个视觉模型的共识来提升特征上采样精度
DiveUp: Learning Feature Upsampling from Diverse Vision Foundation Models
- 引入多模型专家协同，通过结构共识正则化上采样学习
- 在多种下游任务中达到当前最优性能，显著提升像素级理解效果
- 适合需要高精度上采样的研究者和工程师
近期，特征上采样因能有效提升视觉基础模型（VFMs）在像素级理解任务中的表现而受到关注。现有方法通常依赖同一模型的高分辨率特征进行自重建式上采样，但这种单一模型依赖会迫使上采样器过拟合于源模型的位置错位和高范数伪影。为此，本文提出DiveUp，一种打破单模型依赖的新框架，通过引入多视觉基础模型的关联引导机制。DiveUp不采用简单特征融合，而是将多样化的VFMs视为一组专家，利用其结构一致性来正则化上采样器的学习过程，有效防止源模型错误空间结构的传播。为解决不同模型间特征空间不对齐问题，提出通用关系特征表示——局部质心（COM）场，提取内在几何结构，实现跨模型无缝交互。此外，设计了一种尖锐性感知选择策略，评估各模型的空间可靠性，过滤掉高范数伪影，仅在每个局部区域聚合最可靠的专家指导。DiveUp是一个统一的、编码器无关的框架，联合训练的模型可无需重训练即可通用上采样来自多样化VFMs的特征。大量实验表明，DiveUp在多种下游密集预测任务中均达到先进水平，验证了多专家关系引导的有效性。代码与模型已公开：https://github.com/Xiaoqiong-Liu/DiveUp
原文摘要 · Abstract (English)
Recently, feature upsampling has gained increasing attention owing to its effectiveness in enhancing vision foundation models (VFMs) for pixel-level understanding tasks. Existing methods typically rely on high-resolution features from the same foundation model to achieve upsampling via self-reconstruction. However, relying solely on intra-model features forces the upsampler to overfit to the source model's inherent location misalignment and high-norm artifacts. To address this fundamental limitation, we propose DiveUp, a novel framework that breaks away from single-model dependency by introducing multi-VFM relational guidance. Instead of naive feature fusion, DiveUp leverages diverse VFMs as a panel of experts, utilizing their structural consensus to regularize the upsampler's learning process, effectively preventing the propagation of inaccurate spatial structures from the source model. To reconcile the unaligned feature spaces across different VFMs, we propose a universal relational feature representation, formulated as a local center-of-mass (COM) field, that extracts intrinsic geometric structures, enabling seamless cross-model interaction. Furthermore, we introduce a spikiness-aware selection strategy that evaluates the spatial reliability of each VFM, effectively filtering out high-norm artifacts to aggregate guidance from only the most reliable expert at each local region. DiveUp is a unified, encoder-agnostic framework; a jointly-trained model can universally upsample features from diverse VFMs without requiring per-model retraining. Extensive experiments demonstrate that DiveUp achieves state-of-the-art performance across various downstream dense prediction tasks, validating the efficacy of multi-expert relational guidance. Our code and models are available at: https://github.com/Xiaoqiong-Liu/DiveUp
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。