重现实验发现语言引导效果被高估，改进标注能提升自监督模型性能。
Impact of Language Guidance: A Reproducibility Study
- 用语言引导采样图像视图对，提升模型概念相似性理解。
- 修复低质标题后，模型性能显著提升，验证了数据质量关键作用。
- 提出基于可解释性的新评估指标，适合研究自监督学习的学者。
现代深度学习模型需大量数据以达到顶尖性能，但人工标注耗时昂贵且易出错。自监督学习的发展使无需显式标注即可训练大模型成为可能。对比学习是主流范式，如 SimCLR 和 CLIP 依赖图像增强或直接最小化图像与文本间的跨模态损失。Banani 等（2023）提出使用语言引导来采样视图对，声称语言能增强概念相似性，抵消视觉变化影响。我们复现其实验，发现其数据集 RedCaps 包含低质量标题。我们使用现成图像生成模型 BLIP-2 替换原标题，显著提升性能，并设计一种基于可解释性方法的新指标，用于评估自监督模型的语义能力。
原文摘要 · Abstract (English)
Modern deep-learning architectures need large amounts of data to produce state-of-the-art results. Annotating such huge datasets is time-consuming, expensive, and prone to human error. Recent advances in self-supervised learning allow us to train huge models without explicit annotation. Contrastive learning is a popular paradigm in self-supervised learning. Recent works like SimCLR and CLIP rely on image augmentations or directly minimizing cross-modal loss between image and text. Banani et al. (2023) propose to use language guidance to sample view pairs. They claim that language enables better conceptual similarity, eliminating the effects of visual variability. We reproduce their experiments to verify their claims and find that their dataset, RedCaps, contains low-quality captions. We use an off-the-shelf image captioning model, BLIP-2, to replace the captions and improve performance, and we also devise a new metric to evaluate the semantic capabilities of self-supervised models based on interpretability methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。