arXiv:2606.31201cs.CV2026-06中稿 · ECCV

让图像不同区域自动分配学习重点,提升多目标自监督模型性能

ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modeling

论文配图:ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modeling
图 1 · 摘自论文原文
  • 用专家路由机制为每个图像块动态分配损失权重
  • 在ImageNet上线性探测达80.6%,微调精度85.3%
  • 适合需要精细特征学习的视觉下游任务

多目标掩码图像建模(MIM)融合了补丁蒸馏、CLS对齐和像素重建等多种学习信号,但现有方法使用全局标量加权,忽略了图像块间的空间差异。本文提出ExPLoRe(专家块级损失路由),将Soft Mixture of Experts(MoE)的调度权重重用于学习性的块级损失系数。核心机制为损失耦合:允许损失梯度通过调度权重反向传播至路由器,实现内容依赖的个性化聚焦。消融实验表明,当梯度被阻断时性能下降1.6%。在ImageNet-1K上,基于ViT-Base的ExPLoRe在两种目标组合下超越非MoE基线(Token+CLS:k-NN提升0.5%,线性探测提升4.4%;Token+Pixel:k-NN提升2.2%),达到80.6%线性探测准确率和85.3%微调准确率,与已有方法相当。针对下游迁移,设计了适配策略(冻结路由、专家丢弃、冻结注意力),使MoE微调提升1.5%,并在ADE20K上缩小2.5–2.9 mIoU差距,使MoE模型在语义分割上达到或超过非MoE基线。

原文摘要 · Abstract (English)

Multi-objective masked image modeling (MIM) combines complementary learning signals (token distillation, CLS alignment, and pixel reconstruction) but existing methods weight these objectives with global scalars, ignoring spatial heterogeneity across patches. We present ExPLoRe (Expert Patch-Level Loss Routing), which repurposes Soft Mixture of Experts (MoE) dispatch weights as learned, per-patch loss coefficients. The key mechanism is loss-coupling: allowing loss gradients to flow through dispatch weights to the router enables content-dependent specialization, where different patches receive different emphases across objectives. A detach ablation confirms loss-coupling as the core mechanism, degrading performance by 1.6% when gradients are blocked. On ImageNet-1K with ViT-Base, ExPLoRe improves over non-MoE baselines on two objective combinations (Token+CLS: +0.5% k-NN, +4.4% linear probe; Token+Pixel: +2.2% k-NN), achieving 80.6% linear probe and 85.3% finetuning accuracy, competitive with published methods. For downstream transfer, we develop adaptation recipes (Freeze Routing, Expert Dropout, and Freeze Attention) that improve MoE finetuning by +1.5% over the vanilla MoE, and close a 2.5--2.9 mIoU segmentation gap so that MoE models match or exceed non-MoE baselines on ADE20K.

自监督学习视觉建模MoE图像分割

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。