统一跨模态遥感图像翻译,支持任意模态间互转。
Any2Any: Unified Arbitrary Modality Translation for Remote Sensing
- 通过共享潜在空间实现多模态统一翻译,避免成对训练。
- 在14项任务中超越传统方法,零样本泛化表现优异。
- 适配遥感研究者,尤其关注多源数据融合的场景。
多模态遥感影像为同一地理场景提供互补观测,但实际中常存在数据缺失。现有跨模态翻译方法将每对模态视为独立任务,导致复杂度呈二次增长,且难以推广至未见模态组合。本文提出Any2Any,将任意模态翻译建模为对场景共享潜在表示的推断,不同模态被视为同一语义的不完整观测。基于此,构建统一的潜在扩散框架,将异构输入映射至几何对齐的潜在空间,采用共享主干网络进行锚定潜在回归,解耦模态特异性表征学习与语义映射。同时引入轻量级目标特定残差适配器,纠正系统性潜在偏差,且不增加推理开销。为支持稀疏但连通的监督学习,提出RST-1M,首个包含五种传感模态、规模达百万级的遥感数据集,为任意模态翻译提供监督锚点。14项翻译任务实验表明,Any2Any持续优于成对翻译方法,并展现出强零样本泛化能力。代码与模型已开源。
原文摘要 · Abstract (English)
Multi-modal remote sensing imagery provides complementary observations of the same geographic scene, yet such observations are frequently incomplete in practice. Existing cross-modal translation methods treat each modality pair as an independent task, resulting in quadratic complexity and limited generalization to unseen modality combinations. We formulate Any-to-Any translation as inference over a shared latent representation of the scene, where different modalities correspond to partial observations of the same underlying semantics. Based on this formulation, we propose Any2Any, a unified latent diffusion framework that projects heterogeneous inputs into a geometrically aligned latent space. Such structure performs anchored latent regression with a shared backbone, decoupling modality-specific representation learning from semantic mapping. Moreover, lightweight target-specific residual adapters are used to correct systematic latent mismatches without increasing inference complexity. To support learning under sparse but connected supervision, we introduce RST-1M, the first million-scale remote sensing dataset with paired observations across five sensing modalities, providing supervision anchors for any-to-any translation. Experiments across 14 translation tasks show that Any2Any consistently outperforms pairwise translation methods and exhibits strong zero-shot generalization to unseen modality pairs. Code and models are available at https://github.com/MiliLab/Any2Any.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。