学习是不可逆过程,会产生成本,用热力学解释为何能从数据中抽象出新知识。
A Thermodynamic Theory of Learning I: Irreversible Ensemble Transport and Epistemic Costs
- 将学习视为概率分布空间中的传输过程,引入认知自由能框架。
- 发现有限时间内实现学习必须产生最小熵,由初始与终态分布的Wasserstein距离决定。
- 提出认知速度极限,适用于所有学习算法,揭示抽象的本质代价。
学习系统从数据中获得结构化内部表征,但经典信息论表明确定性变换不会增加信息。这引发根本问题:学习如何在不违反信息论限制的情况下产生抽象与洞察?我们论证,有限时间内进行的学习本质上是不可逆的,而认知结构的实现必然伴随熵产生。为此,我们将学习建模为模型配置概率分布空间中的传输过程,并引入认知自由能框架。在此框架中,定义自由能减少为记录学习轨迹上总认知自由能下降的账目量。该表述表明,在有限时间内实现此减少必然导致不可逆熵产生。随后推导出认知速度极限(ESL),一个有限时间不等式,下界给出了任意学习过程实现给定分布变换所需的最小熵生产。该下界仅依赖于初始与最终系综分布间的Wasserstein距离,且与具体学习算法无关。
原文摘要 · Abstract (English)
Learning systems acquire structured internal representations from data, yet classical information-theoretic results state that deterministic transformations do not increase information. This raises a fundamental question: how can learning produce abstraction and insight without violating information-theoretic limits? We argue that learning is inherently an irreversible process when performed over finite time, and that the realization of epistemic structure necessarily incurs entropy production. To formalize this perspective, we model learning as a transport process in the space of probability distributions over model configurations and introduce an epistemic free-energy framework. Within this framework, we define the free-energy reduction as a bookkeeping quantity that records the total reduction of epistemic free energy along a learning trajectory. This formulation highlights that realizing such a reduction over finite time necessarily incurs irreversible entropy production. We then derive the Epistemic Speed Limit (ESL), a finite-time inequality that lower-bounds the minimal entropy production required by any learning process to realize a given distributional transformation. This bound depends only on the Wasserstein distance between initial and final ensemble distributions and is independent of the specific learning algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。