通过对齐嵌入空间提升预训练编码器的分布外泛化能力
Improving OOD Generalization of Pre-trained Encoders via Aligned Embedding-Space Ensembles
- 无监督对齐多个编码器的嵌入空间，构建有效集成
- 在MNIST上显著提升分布内与分布外数据的嵌入质量
- 适合关注模型泛化性、少样本场景的研究者
未经微调的自监督预训练嵌入在分布外（OOD）数据上的表现较差。一种简单有效的提升预训练表示在OOD数据上泛化能力的方法是使用深度集成。然而，在仅使用无标签数据的情况下，如何获得有效的嵌入空间集成仍是未解问题。我们首先进行理论分析，揭示了集成中各超球形嵌入空间之间的关系，随后设计了一种无监督的、原则性的方法来对齐这些嵌入空间。在MNIST数据集上的实验结果表明，我们的嵌入空间集成方法相较于单个编码器，在分布内和分布外数据上均提升了预训练嵌入的质量。
原文摘要 · Abstract (English)
The quality of self-supervised pre-trained embeddings on out-of-distribution (OOD) data is poor without fine-tuning. A straightforward and simple approach to improving the generalization of pre-trained representation to OOD data is the use of deep ensembles. However, obtaining an effective ensemble in the embedding space with only unlabeled data remains an unsolved problem. We first perform a theoretical analysis that reveals the relationship between individual hyperspherical embedding spaces in an ensemble. We then design a principled method to align these embedding spaces in an unsupervised manner. Experimental results on the MNIST dataset show that our embedding-space ensemble method improves pre-trained embedding quality on in-distribution and OOD data compared to single encoders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。