通过低维约束提升自监督嵌入熵，仅用少量训练就能显著改善下游任务表现。
Improving Pre-trained Self-Supervised Embeddings Through Effective Entropy Maximization
- 设计可高效估计的低维熵约束，替代高维复杂熵估计。
- 在已有模型上继续训练数个周期，下游性能稳定提升，部分任务显著改善。
- 适用于希望微调预训练模型以增强泛化能力的研究者或工程师。
自监督学习（SSL）旨在为未知的轻量级下游任务生成最优预训练嵌入。其中一种策略是最大化嵌入在紧凑空间中的熵。然而，传统熵最大化常依赖于高维熵估计，而高维下估计效果通常较差。本文提出有效熵最大化准则（E2MC），基于易于估计的低维约束。实验表明，仅用数个训练周期对已训练的SSL模型继续预训练，即可持续且显著提升下游性能。消融实验验证了性能提升源于该新增准则。相比之下，使用其他预训练准则无法带来明显改进，甚至可能降低性能。
原文摘要 · Abstract (English)
A number of different architectures and loss functions have been applied to the problem of self-supervised learning (SSL), with the goal of developing embeddings that provide the best possible pre-training for as-yet-unknown, lightly supervised downstream tasks. One of these SSL criteria is to maximize the entropy of a set of embeddings in some compact space. But the goal of maximizing the embedding entropy often depends -- whether explicitly or implicitly -- upon high dimensional entropy estimates, which typically perform poorly in more than a few dimensions. In this paper, we motivate an effective entropy maximization criterion (E2MC), defined in terms of easy-to-estimate, low-dimensional constraints. We demonstrate that using it to continue training an already-trained SSL model for only a handful of epochs leads to a consistent and, in some cases, significant improvement in downstream performance. We perform careful ablation studies to show that the improved performance is due to the proposed add-on criterion. We also show that continued pre-training with alternative criteria does not lead to notable improvements, and in some cases, even degrades performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。