提出可处理时变输入的新型神经网络训练方法,与硬件友好型算法等价。
Lagrangian-based Equilibrium Propagation: generalisation to arbitrary boundary conditions & equivalence with Hamiltonian Echo Learning
- 基于拉格朗日框架扩展能量模型学习,支持任意边界条件
- 证明哈密顿回声学习是该框架的特例,具备高效前向训练特性
- 适合追求低功耗、局部更新的类脑硬件实现
平衡传播(EP)是一种用于静态输入下能量模型训练的算法,依赖于其固定点的变分描述。将EP推广到时变输入是一项挑战,因为变分描述需适用于整个系统轨迹,且边界条件需谨慎处理。本文提出广义拉格朗日平衡传播(GLEP),将EP的变分形式扩展至时变输入。我们证明,根据系统边界条件的不同,GLEP会导出不同的学习算法,其中许多不适用于实际实现。随后我们证明,哈密顿回声学习(HEL)——包括最近提出的递归HEL(RHEL)和早期的哈密顿回声反向传播(HEB)——可作为GLEP的一个特例。值得注意的是,只有HEL继承了使EP成为反向传播替代方案的特性:它采用‘仅前向’运行模式(即推理与学习使用同一系统)、计算高效(无论模型规模大小,仅需两次或多次通过系统)并支持局部学习。
原文摘要 · Abstract (English)
Equilibrium Propagation (EP) is a learning algorithm for training Energy-based Models (EBMs) on static inputs which leverages the variational description of their fixed points. Extending EP to time-varying inputs is a challenging problem, as the variational description must apply to the entire system trajectory rather than just fixed points, and careful consideration of boundary conditions becomes essential. In this work, we present Generalized Lagrangian Equilibrium Propagation (GLEP), which extends the variational formulation of EP to time-varying inputs. We demonstrate that GLEP yields different learning algorithms depending on the boundary conditions of the system, many of which are impractical for implementation. We then show that Hamiltonian Echo Learning (HEL) -- which includes the recently proposed Recurrent HEL (RHEL) and the earlier known Hamiltonian Echo Backpropagation (HEB) algorithms -- can be derived as a special case of GLEP. Notably, HEL is the only instance of GLEP we found that inherits the properties that make EP a desirable alternative to backpropagation for hardware implementations: it operates in a "forward-only" manner (i.e. using the same system for both inference and learning), it scales efficiently (requiring only two or more passes through the system regardless of model size), and enables local learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。