研究遥感多模态数据对齐如何保留特有信息，发现对齐可能造成信息丢失。
Can multimodal representation learning by alignment preserve modality-specific information?
- 通过理论分析揭示对齐策略在特定条件下会引发信息损失。
- 实验验证在真实场景中对齐仍可能削弱非共享模态的有用信息。
- 适合关注多源遥感融合与对比学习改进的研究者参考。
多模态数据融合是众多机器学习任务的关键问题，尤其在地球观测领域。早期方法依赖特定神经网络架构和监督学习，但标签数据稀缺促使自监督学习的发展。当前先进方法利用不同模态卫星数据在相同地理区域的空间对齐，以促进潜在空间中的语义对齐。本文研究此类方法是否能保留跨模态不共享的任务相关特征。首先，在简化假设下证明对齐策略本质上可能导致信息损失；随后通过更贴近现实的数值实验支持该理论发现。这些理论与实证结果旨在推动面向多模态卫星数据的对比学习新方法发展。代码与数据公开于 https://github.com/Romain3Ch216/alg_maclean_25。
原文摘要 · Abstract (English)
Combining multimodal data is a key issue in a wide range of machine learning tasks, including many remote sensing problems. In Earth observation, early multimodal data fusion methods were based on specific neural network architectures and supervised learning. Ever since, the scarcity of labeled data has motivated self-supervised learning techniques. State-of-the-art multimodal representation learning techniques leverage the spatial alignment between satellite data from different modalities acquired over the same geographic area in order to foster a semantic alignment in the latent space. In this paper, we investigate how this methods can preserve task-relevant information that is not shared across modalities. First, we show, under simplifying assumptions, when alignment strategies fundamentally lead to an information loss. Then, we support our theoretical insight through numerical experiments in more realistic settings. With those theoretical and empirical evidences, we hope to support new developments in contrastive learning for the combination of multimodal satellite data. Our code and data is publicly available at https://github.com/Romain3Ch216/alg_maclean_25.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。