提出新方法实现多模态数据解耦，提升可解释性与下游任务性能。
An Information Criterion for Controlled Disentanglement of Multimodal Data
- 基于自监督学习构建解耦表示，聚焦无法达到最小必要信息点的场景
- 在合成与真实数据集上成功分离共享与模态特有特征，优于基线模型
- 适用于视觉语言预测和生物分子表型检索等任务，适合多模态研究者
多模态表征学习旨在关联并分解多种模态中的内在信息。通过将模态特有信息与跨模态共享信息解耦，可增强可解释性与鲁棒性，并支持反事实结果生成等下游任务。然而，在许多实际应用中，两类信息往往深度耦合，分离难度大。本文提出解耦自监督学习（DisentangledSSL），一种新型自监督方法以学习解耦表征。我们对每种解耦表示的最优性进行了全面分析，尤其关注先前研究未覆盖的情形——即所谓的最小必要信息（MNI）点不可达的情况。实验表明，DisentangledSSL 在多个合成与真实世界数据集上均能有效学习共享与模态特有特征，并在各类下游任务中持续优于基线模型，包括视觉-语言数据的预测任务以及生物数据中的分子-表型检索任务。代码已公开于 https://github.com/uhlerlab/DisentangledSSL。
原文摘要 · Abstract (English)
Multimodal representation learning seeks to relate and decompose information inherent in multiple modalities. By disentangling modality-specific information from information that is shared across modalities, we can improve interpretability and robustness and enable downstream tasks such as the generation of counterfactual outcomes. Separating the two types of information is challenging since they are often deeply entangled in many real-world applications. We propose Disentangled Self-Supervised Learning (DisentangledSSL), a novel self-supervised approach for learning disentangled representations. We present a comprehensive analysis of the optimality of each disentangled representation, particularly focusing on the scenario not covered in prior work where the so-called Minimum Necessary Information (MNI) point is not attainable. We demonstrate that DisentangledSSL successfully learns shared and modality-specific features on multiple synthetic and real-world datasets and consistently outperforms baselines on various downstream tasks, including prediction tasks for vision-language data, as well as molecule-phenotype retrieval tasks for biological data. The code is available at https://github.com/uhlerlab/DisentangledSSL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。