将多模态特征拆分为单一与交互部分，提升学习效率与任务适应性。
Partitioner Guided Modal Learning Framework
- 通过分区器分离单模态与跨模态特征，分别优化学习。
- 在四个任务中表现优异，支持不同学习率与分布调整。
- 适合需要灵活多模态融合的下游应用，如跨模态检索与生成。
多模态学习得益于多种模态信息，每种模态表示可划分为仅来自单模态训练的单模态特征，以及通过跨模态交互学习的配对模态特征。基于此视角，我们提出分区引导的多模态学习框架 PgM，包含模态分区器、单模态学习器、配对模态学习器和单-配对模态解码器。模态分区器将学习到的模态表示分割为单模态与配对模态特征；模态学习器包含两个专用组件，分别用于单模态和配对模态学习；单-配对模态解码器基于单模态和配对模态特征重构模态表示。PgM 具有三大优势：1）全面学习单模态与配对模态特征；2）灵活调整单模态与配对模态表示的分布，以适配多样下游任务；3）支持跨模态与分区间不同学习率。大量实验验证了 PgM 在四项多模态任务中的有效性，并进一步展示了其对现有模型的可迁移性。此外，我们可视化了不同模态与任务下单模态与配对模态特征的分布，揭示了它们各自的贡献。
原文摘要 · Abstract (English)
Multimodal learning benefits from multiple modal information, and each learned modal representations can be divided into uni-modal that can be learned from uni-modal training and paired-modal features that can be learned from cross-modal interaction. Building on this perspective, we propose a partitioner-guided modal learning framework, PgM, which consists of the modal partitioner, uni-modal learner, paired-modal learner, and uni-paired modal decoder. Modal partitioner segments the learned modal representation into uni-modal and paired-modal features. Modal learner incorporates two dedicated components for uni-modal and paired-modal learning. Uni-paired modal decoder reconstructs modal representation based on uni-modal and paired-modal features. PgM offers three key benefits: 1) thorough learning of uni-modal and paired-modal features, 2) flexible distribution adjustment for uni-modal and paired-modal representations to suit diverse downstream tasks, and 3) different learning rates across modalities and partitions. Extensive experiments demonstrate the effectiveness of PgM across four multimodal tasks and further highlight its transferability to existing models. Additionally, we visualize the distribution of uni-modal and paired-modal features across modalities and tasks, offering insights into their respective contributions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。