用自生成图像增强提升自监督学习,无需文本标注。
Gen-SIS: Generative Self-augmentation Improves Self-supervised Learning
- 用自编码器嵌入引导扩散模型生成多样图像视图。
- 在自然图像和病理图像上均提升下游任务性能。
- 适合无文本数据的医学等专业领域使用。
自监督学习(SSL)通过最大化同一图像不同视图特征的相似性来训练图像编码器。现有方法依赖人工设计的增强手段(如随机裁剪、色彩抖动)生成视图。近期,生成式扩散模型通过提供更丰富的数据增强提升了SSL效果,但这些模型通常需在大规模图文数据集上预训练,难以应用于病理学等专业领域。本文提出Gen-SIS,一种仅基于无标签图像数据训练的扩散增强技术,完全避免对文本描述等外部监督的依赖。首先使用人工增强训练初始SSL编码器;随后以该编码器的嵌入作为条件,训练一个扩散模型。训练后,给定源图像的嵌入,该模型可生成其多样化视图。我们证明,这些“自增强”(self-augmentations)有助于训练更强的SSL编码器。此外,基于编码器隐空间的插值能力,我们提出一种新预训练任务:解耦合成图像的两个原始来源图像。在自然图像与数字病理图像上,Gen-SIS均在多个下游任务中取得性能提升。
原文摘要 · Abstract (English)
Self-supervised learning (SSL) methods have emerged as strong visual representation learners by training an image encoder to maximize similarity between features of different views of the same image. To perform this view-invariance task, current SSL algorithms rely on hand-crafted augmentations such as random cropping and color jittering to create multiple views of an image. Recently, generative diffusion models have been shown to improve SSL by providing a wider range of data augmentations. However, these diffusion models require pre-training on large-scale image-text datasets, which might not be available for many specialized domains like histopathology. In this work, we introduce Gen-SIS, a diffusion-based augmentation technique trained exclusively on unlabeled image data, eliminating any reliance on external sources of supervision such as text captions. We first train an initial SSL encoder on a dataset using only hand-crafted augmentations. We then train a diffusion model conditioned on embeddings from that SSL encoder. Following training, given an embedding of the source image, this diffusion model can synthesize its diverse views. We show that these `self-augmentations', i.e. generative augmentations based on the vanilla SSL encoder embeddings, facilitate the training of a stronger SSL encoder. Furthermore, based on the ability to interpolate between images in the encoder latent space, we introduce the novel pretext task of disentangling the two source images of an interpolated synthetic image. We validate Gen-SIS's effectiveness by demonstrating performance improvements across various downstream tasks in both natural images, which are generally object-centric, as well as digital histopathology images, which are typically context-based.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。