arXiv:2503.19331cs.CVcs.LG2025-03NeurIPS被引 10

针对多通道图像重建难题,提出新方法提升跨通道信息利用。

ChA-MAEViT: Unifying Channel-Aware Masked Autoencoders and Multi-Channel Vision Transformers for Improved Cross-Channel Learning

  • 动态通道-块掩码,强制模型学习跨通道关联
  • 在卫星与显微图像上提升3.0%-21.5%性能
  • 适合遥感、医学图像等多通道视觉任务

以往基于掩码自编码器(MAE)的方法依赖随机块掩码,假设图像各通道间存在冗余,可利用跨通道相关性重建被遮挡内容。但在多通道成像(MCI)中,通道常提供互补信息且重叠极少,导致现有MAE主要学习单通道局部结构,未能充分挖掘跨通道交互,限制了其在MCI中的表现。本文提出ChA-MAEViT,通过四种策略增强多通道学习:(1) 动态通道-块掩码,迫使模型重建缺失通道与被遮块,强化跨通道依赖并提升对不同通道配置的鲁棒性;(2) 引入记忆令牌,作为长期记忆辅助跨通道信息共享,缓解结构差异带来的重建挑战;(3) 混合令牌融合模块,将细粒度块令牌与全局类别令牌融合,捕获更丰富表示;(4) 通道感知解码器,使用通道令牌轻量高效重建图像块。在卫星与显微图像数据集CHAMMI、JUMP-CP和So2Sat上的实验表明,ChA-MAEViT显著优于当前最优的MCI-ViTs,性能提升达3.0%-21.5%,凸显跨通道交互在MCI中的关键作用。代码已开源于https://github.com/chaudatascience/cha_mae_vit。

原文摘要 · Abstract (English)

Prior work using Masked Autoencoders (MAEs) typically relies on random patch masking based on the assumption that images have significant redundancies across different channels, allowing for the reconstruction of masked content using cross-channel correlations. However, this assumption does not hold in Multi-Channel Imaging (MCI), where channels may provide complementary information with minimal feature overlap. Thus, these MAEs primarily learn local structures within individual channels from patch reconstruction, failing to fully leverage cross-channel interactions and limiting their MCI effectiveness. In this paper, we present ChA-MAEViT, an MAE-based method that enhances feature learning across MCI channels via four key strategies: (1) dynamic channel-patch masking, which compels the model to reconstruct missing channels in addition to masked patches, thereby enhancing cross-channel dependencies and improving robustness to varying channel configurations; (2) memory tokens, which serve as long-term memory aids to promote information sharing across channels, addressing the challenges of reconstructing structurally diverse channels; (3) hybrid token fusion module, which merges fine-grained patch tokens with a global class token to capture richer representations; and (4) Channel-Aware Decoder, a lightweight decoder utilizes channel tokens to effectively reconstruct image patches. Experiments on satellite and microscopy datasets, CHAMMI, JUMP-CP, and So2Sat, show that ChA-MAEViT significantly outperforms state-of-the-art MCI-ViTs by 3.0-21.5%, highlighting the importance of cross-channel interactions in MCI. Our code is publicly available at https://github.com/chaudatascience/cha_mae_vit.

多通道图像自编码器视觉Transformer遥感图像

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。