arXiv:2505.06635cs.CV2025-05ICCV被引 12

用熵正则化让多模态分割模型不偏科,提升鲁棒性。

Reducing Unimodal Bias in Multi-Modal Semantic Segmentation with Multi-Scale Functional Entropy Regularization

  • 引入无参数的多尺度熵正则项,平衡各模态贡献。
  • 在三个数据集上分别提升13.94%、3.25%、3.64%。
  • 适合传感器失效场景下的多模态分割任务。

为密集预测任务(尤其是语义分割)融合并平衡来自新型传感器的多模态输入至关重要,但目前仍面临显著挑战。主要问题在于多模态框架易过度依赖易学习的模态,即出现单模态主导或偏差。这在真实场景中尤为严重——当主导模态不可用时,性能会大幅下降。为此,我们提出一种简单而有效的无参插件式熵正则化方法,通过功能熵与功能费雪信息的对偶关系,最大化各视觉模态的信息贡献,从而缓解单模态主导现象,建立更均衡、鲁棒的分割框架。进一步设计多尺度正则模块,在高层特征和分割预测上施加该正则项,实现更全面的平衡。在三个数据集上的大量实验表明,该方法在不引入任何额外参数的情况下,性能分别提升13.94%、3.25%和3.64%。

原文摘要 · Abstract (English)

Fusing and balancing multi-modal inputs from novel sensors for dense prediction tasks, particularly semantic segmentation, is critically important yet remains a significant challenge. One major limitation is the tendency of multi-modal frameworks to over-rely on easily learnable modalities, a phenomenon referred to as unimodal dominance or bias. This issue becomes especially problematic in real-world scenarios where the dominant modality may be unavailable, resulting in severe performance degradation. To this end, we apply a simple but effective plug-and-play regularization term based on functional entropy, which introduces no additional parameters or modules. This term is designed to intuitively balance the contribution of each visual modality to the segmentation results. Specifically, we leverage the log-Sobolev inequality to bound functional entropy using functional-Fisher-information. By maximizing the information contributed by each visual modality, our approach mitigates unimodal dominance and establishes a more balanced and robust segmentation framework. A multi-scale regularization module is proposed to apply our proposed plug-and-play term on high-level features and also segmentation predictions for more balanced multi-modal learning. Extensive experiments on three datasets demonstrate that our proposed method achieves superior performance, i.e., +13.94%, +3.25%, and +3.64%, without introducing any additional parameters.

多模态语义分割熵正则不平衡

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。