CLIPin提升图文对齐，无需对比学习即可增强模型泛化能力
CLIPin: A Non-contrastive Plug-in to CLIP for Multimodal Semantic Alignment
- 提出非对比式插件CLIPin，无缝集成到CLIP架构中
- 在多个下游任务上显著提升图文对齐效果，兼容多种对比框架
- 设计共享预投影层，平衡对比与非对比学习，参数开销小
大规模自然图像-文本数据集（尤其网络自动收集）常因弱监督导致语义对齐松散，而医学数据集虽跨模态相关性强但内容多样性低。这些特性给对比语言图像预训练（CLIP）带来挑战：难以学习鲁棒且通用的表示。本文提出CLIPin，一种统一的非对比式插件，可无缝集成至CLIP类架构，提升多模态语义对齐，提供更强监督并增强对齐鲁棒性。此外，为图像和文本模态分别设计两个共享预投影器，以参数高效方式融合对比与非对比学习。大量下游任务实验证明，CLIPin作为即插即用组件，在多种对比框架下均具有效性和通用性。代码已开源。
原文摘要 · Abstract (English)
Large-scale natural image-text datasets, especially those automatically collected from the web, often suffer from loose semantic alignment due to weak supervision, while medical datasets tend to have high cross-modal correlation but low content diversity. These properties pose a common challenge for contrastive language-image pretraining (CLIP): they hinder the model's ability to learn robust and generalizable representations. In this work, we propose CLIPin, a unified non-contrastive plug-in that can be seamlessly integrated into CLIP-style architectures to improve multimodal semantic alignment, providing stronger supervision and enhancing alignment robustness. Furthermore, two shared pre-projectors are designed for image and text modalities respectively to facilitate the integration of contrastive and non-contrastive learning in a parameter-compromise manner. Extensive experiments on diverse downstream tasks demonstrate the effectiveness and generality of CLIPin as a plug-and-play component compatible with various contrastive frameworks. Code is available at https://github.com/T6Yang/CLIPin.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。