用合成负样本提升对比学习效果，无需额外计算。
SynCo: Synthetic Hard Negatives for Contrastive Visual Representation Learning
- 从缓存嵌入中直接生成硬负样本，不需额外前向传播。
- 在ImageNet上比MoCo-v2高0.4%，比MoCHi高1.0%。
- 适合需要高效预训练的视觉模型研究者。
对比学习依赖有信息量的负样本以塑造表示空间，但获取硬负样本成本较高，通常需要大批次或大量内存缓冲区。我们提出SynCo（对比学习中的合成负样本），一种直接从缓存队列嵌入中合成硬负样本的方法，无需额外前向传播或输入空间处理。我们发现六种轻量级合成策略——涵盖几何、随机和对抗性扰动类型——在几乎无计算开销下持续提升表示性能。尽管适用于任何InfoNCE-based对比目标，我们在MoCo框架内验证了SynCo。在ImageNet ILSVRC-2012线性评估中，200轮预训练时，SynCo相较于MoCo-v2提升0.4%，相较于MoCHi提升1.0%。与MoCHi不同，后者在长周期预训练下表现下降（800轮时比MoCo-v2低2.4%），SynCo保持稳定：通过简单合成负样本调度，在800轮时性能比MoCo-v2高0.5%。SynCo在多种下游任务中也表现出良好的迁移能力。
原文摘要 · Abstract (English)
Contrastive learning relies on informative negatives to shape the representation space, yet obtaining hard negatives is costly, often requiring large batch sizes or extensive memory banks. We propose SynCo (Synthetic negatives in Contrastive learning), an approach that synthesizes hard negatives directly in the representation space from cached queue embeddings, with no additional forward passes or input-space processing. We find that six lightweight synthesis strategies, exhaustively covering the geometric, stochastic, and adversarial perturbation families, consistently improve learned representations at negligible computational cost. Although applicable to any InfoNCE-based contrastive objective, we demonstrate SynCo within the MoCo framework. On ImageNet ILSVRC-2012 linear evaluation at 200 epochs, SynCo yields improvements of +0.4% over MoCo-v2 and +1.0% over MoCHi. Unlike MoCHi, which degrades at extended pretraining schedules (underperforming MoCo-v2 by 2.4% at 800 epochs), SynCo does not: with a simple synthetic negative schedule, performance improves by +0.5% over MoCo-v2 at 800 epochs. SynCo also transfers well to a range of downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。