修正了常用互信息估计方法的偏差问题，提出更准确的锚点估计新方法。
Contrastive Predictive Coding Done Right for Mutual Information Estimation
- 引入辅助锚点类，实现一致的概率密度比估计
- 使用对数评分规则的锚点方法误差显著降低
- 适用于需要高精度互信息估计的研究者
InfoNCE目标虽广泛用于互信息（MI）估计，但其与真实MI的关联并不直接。本文揭示了为何InfoNCE不适合作为有效的MI估计器，并提出改进方法——InfoNCE-anchor。该方法通过引入辅助锚点类，实现一致的密度比估计，得到偏差更小的即插即用式MI估计器。进一步地，基于合适的评分规则对框架进行推广，当采用对数评分时可恢复InfoNCE-anchor作为特例。该统一框架涵盖了NCE、InfoNCE及f-散度变体等多种对比目标。实验表明，使用对数评分的InfoNCE-anchor在互信息估计上最精确；但在自监督表示学习任务中，加入锚点并未提升下游性能。这表明对比学习的优势并非源于精确的互信息估计，而在于结构化密度比的学习。
原文摘要 · Abstract (English)
The InfoNCE objective, originally introduced for contrastive representation learning, has become a popular choice for mutual information (MI) estimation, despite its indirect connection to MI. In this paper, we demonstrate why InfoNCE should not be regarded as a valid MI estimator, and we introduce a simple modification, which we refer to as InfoNCE-anchor, for accurate MI estimation. Our modification introduces an auxiliary anchor class, enabling consistent density ratio estimation and yielding a plug-in MI estimator with significantly reduced bias. Beyond this, we generalize our framework using proper scoring rules, which recover InfoNCE-anchor as a special case when the log score is employed. This formulation unifies a broad spectrum of contrastive objectives, including NCE, InfoNCE, and $f$-divergence variants, under a single principled framework. Empirically, we find that InfoNCE-anchor with the log score achieves the most accurate MI estimates; however, in self-supervised representation learning experiments, we find that the anchor does not improve the downstream task performance. These findings corroborate that contrastive representation learning benefits not from accurate MI estimation per se, but from the learning of structured density ratios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。