提出轻量级跨模态对比学习框架，小批量下仍具强鲁棒性。
AmCLR: Unified Augmented Learning for Cross-Modal Representations
- 融合文本重写与图像变换增强对齐
- 仅需几百样本批量，性能超越需3万样本的CLIP
- 适合资源受限场景下的多模态模型训练
对比学习已成为表示学习的核心框架，推动了单模态与双模态应用的发展。为解决大规模批量依赖和双模态局限性问题，本文受SogCLR启发，提出AmCLR与xAmCLR目标函数，专用于视觉-语言双模态模型。AmCLR整合多种数据增强（如文本改写、图像变换），强化对比表示对齐，将批量大小限制在数百样本内，显著低于CLIP所需的32,768。xAmCLR进一步引入原模态与增强模态间的模态内对齐，提升特征表达能力。该框架在保持计算效率的同时，实现更鲁棒、泛化性更强的对比学习，突破扩展性与增强多样性瓶颈。基于现有SogCLR架构，实验证明其以更少资源获得更优表示质量，为可扩展、鲁棒的多模态学习奠定基础。
原文摘要 · Abstract (English)
Contrastive learning has emerged as a pivotal framework for representation learning, underpinning advances in both unimodal and bimodal applications like SimCLR and CLIP. To address fundamental limitations like large batch size dependency and bimodality, methods such as SogCLR leverage stochastic optimization for the global contrastive objective. Inspired by SogCLR's efficiency and adaptability, we introduce AmCLR and xAmCLR objective functions tailored for bimodal vision-language models to further enhance the robustness of contrastive learning. AmCLR integrates diverse augmentations, including text paraphrasing and image transformations, to reinforce the alignment of contrastive representations, keeping batch size limited to a few hundred samples unlike CLIP which needs batch size of 32,768 to produce reasonable results. xAmCLR further extends this paradigm by incorporating intra-modal alignments between original and augmented modalities for richer feature learning. These advancements yield a more resilient and generalizable contrastive learning process, aimed at overcoming bottlenecks in scaling and augmentative diversity. Since we have built our framework on the existing SogCLR, we are able to demonstrate improved representation quality with fewer computational resources, establishing a foundation for scalable and robust multi-modal learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。