用重整化群方法构建分层最大熵模型,统一多层级系统建模。
Hierarchical Maximum Entropy via the Renormalization Group
- 基于重整化群思想,提出分层最大熵框架
- 通过参数流实现多层级熵最大化,简化计算
- 适用于物理与机器学习中的分层建模场景
分层结构在统计模型、机器学习及物理系统中普遍存在。在均值约束下,最大熵分布为指数形式的Gibbs-Boltzmann分布,我们由此拓展出‘分层最大熵’框架,用于处理多层级模型。通过构建吉布斯变分原理和Donsker-Varadhan熵变分表示的多层级扩展,我们证明:在所有层级变换下熵最大的帕累托最优分布,可通过理论物理中的重整化群程序获得。此外,我们研究了具有分层对称性的场景,显著简化重整化群过程,包括二次模损失函数、对数损失函数及最近邻损失函数。这通过引入‘参数流’概念实现,其作用类似重整化群理论中的重整化流。本工作连接了概率论、信息论与统计力学的思想。
原文摘要 · Abstract (English)
Hierarchical structures, which include multiple levels, are prevalent in statistical and machine-learning models as well as physical systems. Extending the foundational result that the maximum entropy distribution under mean constraints is given by the exponential Gibbs-Boltzmann form, we introduce the framework of "hierarchical maximum entropy" to address these multilevel models. We demonstrate that Pareto optimal distributions, which maximize entropies across all levels of hierarchical transformations, can be obtained via renormalization-group procedures from theoretical physics. This is achieved by formulating multilevel extensions of the Gibbs variational principle and the Donsker-Varadhan variational representation of entropy. Moreover, we explore settings with hierarchical invariances that significantly simplify the renormalization-group procedures, enhancing computational efficiency: quadratic modular loss functions, logarithmic loss functions, and nearest-neighbor loss functions. This is accomplished through the introduction of the concept of parameter flows, which serves as an analog to renormalization flows in renormalization group theory. This work connects ideas from probability theory, information theory, and statistical mechanics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。