针对多模态模型在分布偏移下的鲁棒性问题,提出自适应高斯校准方法。
Multi-modal Test-time Adaptation via Adaptive Probabilistic Gaussian Calibration

- 设计自适应概率高斯模型,显式建模类别条件分布
- 在多个基准上实现当前最优性能,跨分布偏移效果稳定
- 适合需要提升推理阶段鲁棒性的多模态系统开发者
多模态测试时自适应(TTA)通过利用推理阶段的无标签目标数据,增强基准多模态模型对分布偏移的鲁棒性。尽管已取得显著成效,但多模态TTA方法的发展受限于缺乏对类别条件分布的显式建模,而该建模对准确预测和可靠决策边界至关重要。经典高斯判别分析(GDA)虽能在单模态场景中提供基础建模并取得适度进展,但在多模态TTA中,模态分布不对称性削弱了其建模能力。为此,我们提出一种专用于多模态TTA的概率高斯模型,显式建模类别条件分布,并进一步设计自适应对比不对称矫正技术,以缓解模态不对称带来的负面影响,从而获得校准预测与可靠决策边界。在多种基准上的大量实验表明,该方法在广泛分布偏移下均达到当前最优性能。代码已公开于 https://github.com/XuJinglinn/AdaPGC。
原文摘要 · Abstract (English)
Multi-modal test-time adaptation (TTA) enhances the resilience of benchmark multi-modal models against distribution shifts by leveraging the unlabeled target data during inference. Despite the documented success, the advancement of multi-modal TTA methodologies has been impeded by a persistent limitation, i.e., the lack of explicit modeling of category-conditional distributions, which is crucial for yielding accurate predictions and reliable decision boundaries. Canonical Gaussian discriminant analysis (GDA) provides a vanilla modeling of category-conditional distributions and achieves moderate advancement in uni-modal contexts. However, in multi-modal TTA scenario, the inherent modality distribution asymmetry undermines the effectiveness of modeling the category-conditional distribution via the canonical GDA. To this end, we introduce a tailored probabilistic Gaussian model for multi-modal TTA to explicitly model the category-conditional distributions, and further propose an adaptive contrastive asymmetry rectification technique to counteract the adverse effects arising from modality asymmetry, thereby deriving calibrated predictions and reliable decision boundaries. Extensive experiments across diverse benchmarks demonstrate that our method achieves state-of-the-art performance under a wide range of distribution shifts. The code is available at https://github.com/XuJinglinn/AdaPGC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。