用统一信息论框架解释多种表示学习方法，提升无监督图像分类性能。
I-Con: A Unifying Framework for Representation Learning
- 基于信息论构建统一框架，将多种损失函数归为最小化条件分布间KL散度。
- 在ImageNet-1K上实现无监督分类准确率提升8%，超越此前最优结果。
- 可指导设计新损失函数与去偏方法，适用于对比学习、聚类等任务。
随着表示学习的发展，针对不同问题出现了大量不同的损失函数。本文提出一个统一的信息论方程，可涵盖机器学习中众多现代损失函数。特别地，我们构建的框架表明，多种主流机器学习方法本质上是在最小化监督表示与学习表示之间条件分布的联合KL散度。这一视角揭示了聚类、谱方法、降维、对比学习与监督学习背后的隐藏信息几何结构。该框架支持融合文献中成功技术，开发新型损失函数。我们不仅证明了23种不同方法之间的关联，还基于此理论构建了达到当前最优水平的无监督图像分类器，在ImageNet-1K上比之前最优结果提升8%。同时，I-Con可用于推导有原则性的去偏方法，显著改善对比表示学习的效果。
原文摘要 · Abstract (English)
As the field of representation learning grows, there has been a proliferation of different loss functions to solve different classes of problems. We introduce a single information-theoretic equation that generalizes a large collection of modern loss functions in machine learning. In particular, we introduce a framework that shows that several broad classes of machine learning methods are precisely minimizing an integrated KL divergence between two conditional distributions: the supervisory and learned representations. This viewpoint exposes a hidden information geometry underlying clustering, spectral methods, dimensionality reduction, contrastive learning, and supervised learning. This framework enables the development of new loss functions by combining successful techniques from across the literature. We not only present a wide array of proofs, connecting over 23 different approaches, but we also leverage these theoretical results to create state-of-the-art unsupervised image classifiers that achieve a +8% improvement over the prior state-of-the-art on unsupervised classification on ImageNet-1K. We also demonstrate that I-Con can be used to derive principled debiasing methods which improve contrastive representation learners.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。