arXiv:2607.27710cs.LG2026-07

提出一种新型神经网络方法,精准估算连续变量的归一化互信息。

NMINE: Normalized Mutual Information Neural Estimation

论文配图:NMINE: Normalized Mutual Information Neural Estimation
图 1 · 摘自论文原文
  • 基于MINE与边际熵估计融合,用神经网络学习互信息与边缘熵。
  • 在1到8维高斯数据上,精度显著优于传统KSG基线方法。
  • 适合需要高维连续变量依赖度量的研究者,如分子动力学与可解释机器学习。

互信息是衡量随机变量间统计依赖性的通用指标,能捕捉线性和非线性关系。对于连续多维变量,需从样本中估计互信息。由于互信息无界,其值在不同数据集、维度或应用间不可直接比较。归一化互信息通过转换为归一化依赖得分解决了此问题。近期研究表明其在分子动力学(arXiv:2405.04980)和可解释机器学习(arXiv:2409.16768)中有实际价值,但现有估计器仍受维度和数值稳定性影响(arXiv:2410.07642)。本文提出一种针对连续变量的全神经归一化互信息估计器。该方法结合基于MINE的神经互信息估计(arXiv:1801.04062)与受MI-NEE启发的神经边缘熵估计(arXiv:1905.12957),利用Donsker--Varadhan表示估计互信息,通过学习各边缘分布与均匀参考分布的散度来估计熵。结果表明,该方法提供了对基于k近邻的归一化互信息估计(arXiv:2405.04980)的神经替代方案。在1至8维高斯数据上的实验显示,该估计器相比KSG基线显著提升精度,表明神经估计在连续多维场景下具有广阔前景。

原文摘要 · Abstract (English)

Mutual information is a general measure of statistical dependence that captures both linear and nonlinear relationships between random variables. For continuous and multidimensional variables For continuous multidimensional variables, mutual information must be estimated from samples. Because mutual information is unbounded, its values are not directly comparable across datasets, dimensions, or applications. Normalized mutual information addresses this limitation by converting mutual information into a normalized dependency score. Recent work has demonstrated the practical value of normalized mutual information in applications such as molecular dynamics {arXiv:2405.04980} and interpretable machine learning {arXiv:2409.16768}, but existing estimators remain sensitive to dimensionality and numerical stability {arXiv:2410.07642}. In this paper, we propose a fully neural normalized mutual information estimator for continuous variables. The proposed approach combines a MINE-based neural mutual information estimator {arXiv:1801.04062} with MI-NEE-inspired neural marginal entropy estimators {arXiv:1905.12957}. Mutual information is estimated using the Donsker--Varadhan representation, while marginal entropies are estimated by learning the divergence between each marginal distribution and a uniform reference distribution, from which entropy is recovered. The resulting estimator provides a neural alternative to k-nearest-neighbor-based normalized mutual information estimation {arXiv:2405.04980}. Experiments on Gaussian data from one to eight dimensions show that the proposed estimator improves accuracy over a KSG-based normalized mutual information baseline. These results indicate that neural estimation is a promising direction for normalized dependency measurement in continuous multidimensional settings.

互信息神经网络归一化高维估计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。