用随机神经网络高效估算KL散度，且误差有理论保证。
A Neural Network Algorithm for KL Divergence Estimation with Quantitative Error Bounds
- 采用浅层随机权重神经网络进行散度估计。
- 误差可达 $O(m^{-1/2}+T^{-1/3})$，与样本数和神经元数相关。
- 首次实现可计算误差上界的神经网络散度估计算法。
估计随机变量间的Kullback-Leibler（KL）散度是统计分析中的基础问题。对于连续随机变量，传统信息论估计器在维度和/或样本量增大时性能急剧下降。为缓解此问题，已有多种基于神经网络的方法被提出用于估计KL散度及相关量（如互信息）。现有理论分析表明存在低误差的神经网络参数，但依赖于非构造性近似定理，无法保证算法实际达到低误差。本文提出一种基于浅层神经网络、随机隐藏层权重和偏置（即随机特征方法）的KL散度估计算法。我们证明：以高概率，该算法的估计误差为 $O(m^{-1/2}+T^{-1/3})$，其中 $m$ 为神经元数量，$T$ 既是算法步数也是样本数。
原文摘要 · Abstract (English)
Estimating the Kullback-Leibler (KL) divergence between random variables is a fundamental problem in statistical analysis. For continuous random variables, traditional information-theoretic estimators scale poorly with dimension and/or sample size. To mitigate this challenge, a variety of methods have been proposed to estimate KL divergences and related quantities, such as mutual information, using neural networks. The existing theoretical analyses show that neural network parameters achieving low error exist. However, since they rely on non-constructive neural network approximation theorems, they do not guarantee that the existing algorithms actually achieve low error. In this paper, we propose a KL divergence estimation algorithm using a shallow neural network with randomized hidden weights and biases (i.e. a random feature method). We show that with high probability, the algorithm achieves a KL divergence estimation error of $O(m^{-1/2}+T^{-1/3})$, where $m$ is the number of neurons and $T$ is both the number of steps of the algorithm and the number of samples.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。