用神经熵量化扩散模型存储信息的能力，发现其高效压缩数据。
Neural Entropy
- 引入神经熵衡量扩散模型在训练中存储的信息量。
- 简单图像模型的神经熵显示其能高效压缩大规模结构化数据。
- 适合对生成模型信息机制感兴趣的读者。
我们通过扩散模型的范式探索深度学习与信息论之间的联系。扩散模型通过不完美地恢复数据被扩散为噪声时丢失的信息，将噪声转化为结构化数据。这些信息在训练过程中存储于神经网络中。我们提出一种称为神经熵的度量来量化该信息，它与扩散过程产生的总熵相关。神经熵不仅依赖于数据分布，还取决于扩散过程本身。对几个简单图像扩散模型的神经熵测量表明，它们在压缩大规模结构化数据方面极为高效。
原文摘要 · Abstract (English)
We explore the connection between deep learning and information theory through the paradigm of diffusion models. A diffusion model converts noise into structured data by reinstating, imperfectly, information that is erased when data was diffused to noise. This information is stored in a neural network during training. We quantify this information by introducing a measure called neural entropy, which is related to the total entropy produced by diffusion. Neural entropy is a function of not just the data distribution, but also the diffusive process itself. Measurements of neural entropy on a few simple image diffusion models reveal that they are extremely efficient at compressing large ensembles of structured data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。