arXiv:2511.18945cs.LGcs.IT2025-11被引 1

用神经网络直接学互信息,速度快且更准。

MIST: Mutual Information Estimation Via Supervised Training

  • 用神经网络端到端训练,直接预测互信息值。
  • 在62.5万组合成数据上训练,跨样本量和维度表现优。
  • 输出置信区间更可靠,适合嵌入复杂学习流程。

我们提出一种完全数据驱动的互信息(MI)估计方法。由于任何MI估计器都是两个随机变量观测样本的函数,我们用神经网络(MIST)参数化该函数,并在包含62.5万个已知真实互信息的合成联合分布的元数据集上进行端到端训练。为处理可变样本量和维度,采用二维注意力机制实现输入样本的置换不变性。为量化不确定性,优化分位数回归损失,使估计器能逼近MI的采样分布而非仅返回点估计。该方法摒弃传统理论保证,换取灵活性与效率。实验表明,所学估计器在不同样本量和维度下均显著优于经典基线,包括训练中未见的分布。基于分位数的置信区间校准良好,比自助法更可靠,且推理速度远超现有神经基线。此外,该框架产出可微、可训练的估计器,可嵌入更大学习流程。利用MI对可逆变换的不变性,通过归一化流可将元数据集适配任意数据模态,支持多样化目标分布的灵活训练。

原文摘要 · Abstract (English)

We propose a fully data-driven approach to designing mutual information (MI) estimators. Since any MI estimator is a function of the observed sample from two random variables, we parameterize this function with a neural network (MIST) and train it end-to-end to predict MI values. Training is performed on a large meta-dataset of 625,000 synthetic joint distributions with known ground-truth MI. To handle variable sample sizes and dimensions, we employ a two-dimensional attention scheme ensuring permutation invariance across input samples. To quantify uncertainty, we optimize a quantile regression loss, enabling the estimator to approximate the sampling distribution of MI rather than return a single point estimate. This research program departs from prior work by taking a fully empirical route, trading universal theoretical guarantees for flexibility and efficiency. Empirically, the learned estimators largely outperform classical baselines across sample sizes and dimensions, including on joint distributions unseen during training. The resulting quantile-based intervals are well-calibrated and more reliable than bootstrap-based confidence intervals, while inference is orders of magnitude faster than existing neural baselines. Beyond immediate empirical gains, this framework yields trainable, fully differentiable estimators that can be embedded into larger learning pipelines. Moreover, exploiting MI's invariance to invertible transformations, meta-datasets can be adapted to arbitrary data modalities via normalizing flows, enabling flexible training for diverse target meta-distributions.

互信息神经网络估计可微

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。