arXiv:2503.02616cs.LGcs.CV2025-03ICLR被引 16

提出SuMi方法,让模型在多模态噪声下测试时仍能稳定适应。

Smoothing the Shift: Towards Stable Test-Time Adaptation under Complex Multimodal Noises

  • 用四分位距平滑避免突变分布干扰,提升适应稳定性。
  • 通过低熵样本筛选与跨模态信息共享,提升多模态特征利用效率。
  • 适合真实场景中复杂噪声混合的多模态模型自适应任务。

测试时自适应(TTA)旨在利用无标签测试数据应对分布偏移,但面对多模态数据时,噪声模式更复杂,如多模态同时损坏或模态缺失,且现实应用中不同分布偏移常混合出现。现有方法因突变分布破坏源模型先验知识而失效。为此,本文提出新挑战——多模态野化TTA。为解决该问题,提出两种新策略:基于四分位距平滑的样本识别与单模态辅助,以及互信息共享(SuMi)。SuMi通过四分位距平滑适应过程,避免突变分布冲击;利用单模态特征筛选低熵、富含多模态信息的样本优化;并通过互信息共享对齐模态间信息,减少差异并提升利用率。在两个公开数据集上的大量实验表明,该方法在复杂噪声模式下显著优于现有方法。代码已开源。

原文摘要 · Abstract (English)

Test-Time Adaptation (TTA) aims to tackle distribution shifts using unlabeled test data without access to the source data. In the context of multimodal data, there are more complex noise patterns than unimodal data such as simultaneous corruptions for multiple modalities and missing modalities. Besides, in real-world applications, corruptions from different distribution shifts are always mixed. Existing TTA methods always fail in such multimodal scenario because the abrupt distribution shifts will destroy the prior knowledge from the source model, thus leading to performance degradation. To this end, we reveal a new challenge named multimodal wild TTA. To address this challenging problem, we propose two novel strategies: sample identification with interquartile range Smoothing and unimodal assistance, and Mutual information sharing (SuMi). SuMi smooths the adaptation process by interquartile range which avoids the abrupt distribution shifts. Then, SuMi fully utilizes the unimodal features to select low-entropy samples with rich multimodal information for optimization. Furthermore, mutual information sharing is introduced to align the information, reduce the discrepancies and enhance the information utilization across different modalities. Extensive experiments on two public datasets show the effectiveness and superiority over existing methods under the complex noise patterns in multimodal data. Code is available at https://github.com/zrguo/SuMi.

测试时自适应多模态稳定学习信息共享

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。