用组合方式激活预训练视觉模型，实现多任务密集预测的高效精准
Learning to Orchestrate Vision Foundation Models for Multi-Task Dense Prediction
- 构建配对复合表征，通过任务引导路由选择最优组合
- 在NYUD-v2和PASCAL-Context上超越单个专家，计算量减半
- 解决路由坍缩问题，适合资源受限下的多任务视觉应用
视觉基础模型（VFMs）因预训练目标不同而具备互补优势。当前多任务密集预测方法仍需训练完整主干网络，或通过额外阶段蒸馏多个模型。本文提出是否可直接利用已有的冻结表示进行下游组合。密集任务需要单一专家无法提供的复合表征。简单融合仅带来微弱提升，而学习路由易坍缩至初始表现强的候选者，使新初始化的组合器缺乏训练信号。为此，我们提出COVE：通过协同组合器构建配对复合候选，并使用任务条件路由在原始与复合候选间分配。为防止坍缩，引入高斯逻辑扰动以增强探索，结合反事实监督，有选择地提升被低估的路由分配。在NYUD-v2和PASCAL-Context数据集上，使用更小的冻结编码器池，计算量约为近期基于VFM方法的一半，仍能匹配或超越基于ViT-L的方法，在多数任务上优于最佳单个冻结专家。
原文摘要 · Abstract (English)
Vision foundation models (VFMs) exhibit complementary strengths shaped by their pretraining objectives. Yet prevailing methods for multi-task dense prediction still train an entire backbone, either by fine-tuning it under multi-task supervision or by distilling multiple VFMs in an additional stage. We ask whether downstream learning can instead compose the frozen representations already available in foundation models. Dense tasks require composite representations that no individual expert provides alone. Realizing them is difficult: simple fusion yields only marginal gains over the best single expert, while learned routing tends to collapse toward candidates that are strong at initialization, starving newly initialized composers of training signal. We present COVE, which constructs pairwise composite candidates through Synergy Composers and routes among raw and composite candidates with a Task-Conditioned Router. To prevent this collapse, COVE combines Gaussian logit perturbation for exploration with counterfactual supervision that selectively increases under-credited routing allocations. On NYUD-v2 and PASCAL-Context, COVE matches or surpasses ViT-L-based methods on most tasks using a smaller frozen encoder pool and roughly half the computation of recent VFM-based competitors, while exceeding the best single frozen expert on every task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。