物理神经网络的计算能耗由内存写入决定,而非算力本身。
Thermodynamic cost of inference and learning in physical neural networks
- 将网络映射为物理哈密顿量,用弹性约束建模层间关系
- 静态推理无能耗,但快速推理耗能与输入差异的平方距离成正比
- 学习参数写入有不可消除的热成本,约每参数几倍k_B T
人工神经网络消耗的能量有多少是由物理规律决定的?对于不可逆数字硬件,参考标准是兰道尔原理:每擦除一位需消耗 $k_B T\ln 2$ 能量。本文将通用前馈网络映射为物理哈密顿量,其中各层关系为弹性相容约束,得出两个精确界:第一,其平衡自由能与输入、所有权重和偏置无关,任意温度下准静态推理无需做功,计算本身无热力学代价;第二,在有限速度下,做功超过初始与最终热态间平方沃瑟斯坦-2距离除以协议时长。若放宽为可区分性熵度量,则热成本等于连续输入间信息差,最快可用速度下约为每维 $k_B T$。学习本质不同:参数写入存在不可消除的成本,每个参数至少几 $k_B T$,且在准静态极限下仍存在。因此,神经网络的热力学代价由其记忆决定,而非算术运算。模拟验证了两个界限:推理功接近传输上限(误差<4%),且一旦耗散功低于热尺度,精度即崩溃。
原文摘要 · Abstract (English)
How much of the energy consumed by artificial neural networks is set by physics rather than by implementation? For irreversible digital hardware the reference is Landauer's principle, which charges $k_B T\ln 2$ per erased bit. We map a generic feedforward network onto a physical Hamiltonian in which each layer relation is an elastic compatibility constraint, and obtain two exact bounds. First, its equilibrium free energy is independent of the input and of every weight and bias, at all temperatures, so quasi-static inference requires no work whatsoever: no thermodynamic cost attaches to computation itself. Second, at finite speed the work exceeds the squared Wasserstein-2 distance between the initial and final thermal states, divided by the protocol duration. Relaxed to an entropic measure of distinguishability, this identifies the cost with the information separating successive inputs, about $k_B T$ per dimension of the widest layer at the fastest usable speed. Learning is fundamentally different: writing the parameters carries an irreducible cost of a few $k_B T$ each that survives the quasi-static limit. The thermodynamic price of a neural network is therefore set by its memory rather than its arithmetic. Simulations confirm both bounds: the inference work saturates the transport bound to within four percent, and accuracy collapses once the dissipated work falls below the thermal scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。