重构目标影响跨域少样本学习效果,新方法提升模型泛化能力。
Reconstruction Target Matters in Masked Image Modeling for Cross-Domain Few-Shot Learning
- 改用特征重建替代像素重建,减少低层域偏见
- 在4个数据集上超越现有方法,实现最新性能
- 适合需要跨域快速适应的视觉模型训练场景
跨域少样本学习(CDFSL)要求模型将数据丰富的源域知识迁移到数据稀缺的目标域以实现快速适应,但显著的域间差异使该任务极具挑战性。掩码自编码器(MAE)擅长利用无标签数据并学习图像全局结构,增强模型泛化能力。然而,在存在显著域偏移的CDFSL任务中,我们发现MAE性能反而低于基线监督模型。本文首次深入分析此现象:MAE在像素重建时倾向于关注低层域特异性信息;若改为重建令牌特征,则可缓解该问题。但并非所有特征都有效——重建高层特征难以提升迁移能力,表明去域信息与保留全局结构之间存在权衡。基于此,我们提出领域无关的掩码图像建模(DAMIM),包含聚合特征重建模块以自动选择重建特征,平衡领域无关信息与图像结构学习,并设计轻量解码器进一步提升编码器泛化能力。在四个CDFSL数据集上的实验表明,该方法达到当前最优性能。
原文摘要 · Abstract (English)
Cross-Domain Few-Shot Learning (CDFSL) requires the model to transfer knowledge from the data-abundant source domain to data-scarce target domains for fast adaptation, where the large domain gap makes CDFSL a challenging problem. Masked Autoencoder (MAE) excels in effectively using unlabeled data and learning image's global structures, enhancing model generalization and robustness. However, in the CDFSL task with significant domain shifts, we find MAE even shows lower performance than the baseline supervised models. In this paper, we first delve into this phenomenon for an interpretation. We find that MAE tends to focus on low-level domain information during reconstructing pixels while changing the reconstruction target to token features could mitigate this problem. However, not all features are beneficial, as we then find reconstructing high-level features can hardly improve the model's transferability, indicating a trade-off between filtering domain information and preserving the image's global structure. In all, the reconstruction target matters for the CDFSL task. Based on the above findings and interpretations, we further propose Domain-Agnostic Masked Image Modeling (DAMIM) for the CDFSL task. DAMIM includes an Aggregated Feature Reconstruction module to automatically aggregate features for reconstruction, with balanced learning of domain-agnostic information and images' global structure, and a Lightweight Decoder module to further benefit the encoder's generalizability. Experiments on four CDFSL datasets demonstrate that our method achieves state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。