无需大量配对数据，逐步将新模态融入统一表示空间。
Continual Cross-Modal Generalization
- 用中介模态逐步将新模态映射到共享离散码本。
- 在多组跨模态任务上表现优异，超越基线方法。
- 适合需要持续扩展模态的多模态系统应用。
跨模态泛化旨在从多模态样本对中学习共享的离散表示空间，实现未标注模态间的知识迁移。然而，为所有模态对构建统一表示需大量配对数据，通常不现实。受 ImageBind 等丰富双模态数据启发，我们探索一种持续学习方法，通过中介模态将新模态逐步映射至共享离散码本。提出持续专家混合适配器（CMoE-Adapter），在保持已有知识的同时将多种模态投影到统一空间。为跨阶段语义对齐，引入动态扩展码本的伪模态回放（PMR）机制，利用已学模态引导新模态适应。在图像-文本、音频-文本、视频-文本和语音-文本等任务上的大量实验表明，该方法在多种跨模态泛化任务中表现强劲。代码已附于补充材料。
原文摘要 · Abstract (English)
Cross-modal generalization aims to learn a shared discrete representation space from multimodal pairs, enabling knowledge transfer across unannotated modalities. However, achieving a unified representation for all modality pairs requires extensive paired data, which is often impractical. Inspired by the availability of abundant bimodal data (e.g., in ImageBind), we explore a continual learning approach that incrementally maps new modalities into a shared discrete codebook via a mediator modality. We propose the Continual Mixture of Experts Adapter (CMoE-Adapter) to project diverse modalities into a unified space while preserving prior knowledge. To align semantics across stages, we introduce a Pseudo-Modality Replay (PMR) mechanism with a dynamically expanding codebook, enabling the model to adaptively incorporate new modalities using learned ones as guidance. Extensive experiments on image-text, audio-text, video-text, and speech-text show that our method achieves strong performance on various cross-modal generalization tasks. Code is provided in the supplementary material.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。