让循环结构可微分,实现用神经网络高效学习迭代算法。
Compiling to recurrent neurons
- 将循环编程编译为线性递归神经元,使其可微分。
- 实验表明,使用该方法的网络学习更快、数据效率更高。
- 适合想融合传统代码逻辑与神经网络的学习者。
离散结构在可微编程中仍处于次要地位,因其函数缺乏显式导数,导致可微程序无法对其求导,限制了其应用范围。例如,神经网络中的条件分支和循环无法随意使用,否则会破坏梯度传播。这并非本质限制。已有研究将条件编译为可微的线性神经元,使条件成为一等公民。本文进一步提出,可通过将循环编译为线性递归神经元,使其同样成为一等公民。我们设计了一种最小化的类型化、高阶、线性的编程语言 Cajal(⊸, 2, N),并证明其程序可正确编译为递归神经元,从而以可微形式表达离散算法。通过实现,在两个实验中将这些递归神经元与神经网络结合,用于解决迭代图像变换任务,预先确定部分功能。结果表明,相比未使用一等循环的网络,该方法学习速度更快,且更具数据效率。关键启示是:递归神经元促进了学习与传统编程离散结构之间的深度互动。
原文摘要 · Abstract (English)
Discrete structures are currently second-class in differentiable programming. Since functions over discrete structures lack overt derivatives, differentiable programs do not differentiate through them and limit where they can be used. For example, when programming a neural network, conditionals and iteration cannot be used everywhere; they can break the derivatives necessary for gradient-based learning to work. This limits the class of differentiable algorithms we can directly express, imposing restraints on how we build neural networks and differentiable programs more generally. However, these restraints are not fundamental. Recent work shows conditionals can be first-class, by compiling them into differentiable form as linear neurons. Similarly, this work shows iteration can be first-class -- by compiling to linear recurrent neurons. We present a minimal typed, higher-order and linear programming language with iteration called $\textsf{Cajal}\scriptstyle(\mathbb{\multimap}, \mathbb{2}, \mathbb{N})$. We prove its programs compile correctly to recurrent neurons, allowing discrete algorithms to be expressed in a differentiable form compatible with gradient-based learning. With our implementation, we conduct two experiments where we link these recurrent neurons against a neural network solving an iterative image transformation task. This determines part of its function prior to learning. As a result, the network learns faster and with greater data-efficiency relative to a neural network programmed without first-class iteration. A key lesson is that recurrent neurons enable a rich interplay between learning and the discrete structures of ordinary programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。