共享编码器提升医疗多模态学习，小数据下更高效
A Shared Encoder Approach to Multimodal Representation Learning
- 用同一组参数处理文本和图像，加可学习模态特征
- 数据少时表现更好，比独立编码器提升显著
- 适合医疗场景中配对数据稀缺的现实应用
多模态表示学习在整合文本与图像等异构数据方面展现出巨大潜力。尽管医疗领域受益于此，但配对数据稀缺及依赖专有或预训练编码器带来挑战。本文提出一种面向医疗领域的共享编码器框架，采用跨模态共享参数，并引入可学习的模态特征。实验证明，该方法优于独立模态编码器，在数据受限情况下表现更优，尤其在训练样本较少时性能提升更明显，凸显其在真实医疗应用中的高效性。代码与实验设置已公开于 https://github.com/VectorInstitute/shared_encoder。
原文摘要 · Abstract (English)
Multimodal representation learning has demonstrated remarkable potential in enabling models to process and integrate diverse data modalities, such as text and images, for improved understanding and performance. While the medical domain can benefit significantly from this paradigm, the scarcity of paired multimodal data and reliance on proprietary or pretrained encoders pose significant challenges. In this work, we present a shared encoder framework for multimodal representation learning tailored to the medical domain. Our approach employs a single set of encoder parameters shared across modalities, augmented with learnable modality features. Empirical results demonstrate that our shared encoder idea achieves superior performance compared to separate modality-specific encoders, demonstrating improved generalization in data-constrained settings. Notably, the performance gains are more pronounced with fewer training examples, underscoring the efficiency of our shared encoder framework for real-world medical applications with limited data. Our code and experiment setup are available at https://github.com/VectorInstitute/shared_encoder.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。