提出可直接最大化互信息的自监督学习方法，突破传统限制。
Explicit Mutual Information Maximization for Self-Supervised Learning
- 利用互信息不变性，在宽松分布假设下实现显式最大化
- 仅用二阶统计量设计损失函数，计算高效且稳定
- 适用于图像等通用数据，尤其适合追求理论严谨性的研究者
自监督学习（SSL）近年来受到广泛关注。理论上，互信息最大化（MIM）是最优准则，具有坚实的信息论基础。然而，由于实际应用中数据分布无法解析表达，难以直接应用MIM。现有方法多为MIM的近似实现。本文表明，在一般分布假设下（即放宽数据分布条件），基于互信息的不变性，可直接应用于SSL。通过分析广义高斯分布验证该结论，并据此仅使用二阶统计量推导出新的损失函数。在多个数据集上进行的大量实验表明，该方法在对比学习和生成模型中均表现优异，有效提升了表征质量。
原文摘要 · Abstract (English)
Recently, self-supervised learning (SSL) has been extensively studied. Theoretically, mutual information maximization (MIM) is an optimal criterion for SSL, with a strong theoretical foundation in information theory. However, it is difficult to directly apply MIM in SSL since the data distribution is not analytically available in applications. In practice, many existing methods can be viewed as approximate implementations of the MIM criterion. This work shows that, based on the invariance property of MI, explicit MI maximization can be applied to SSL under a generic distribution assumption, i.e., a relaxed condition of the data distribution. We further illustrate this by analyzing the generalized Gaussian distribution. Based on this result, we derive a loss function based on the MIM criterion using only second-order statistics. We implement the new loss for SSL and demonstrate its effectiveness via extensive experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。