用线性神经元编译离散算法,让神经网络可直接编程并加速学习。
Compiling to linear neurons
- 设计线性编程语言 Cajal,支持离散算法的可微表达。
- 实验显示连接线性神经元后,模型学习更快、更省数据。
- 适合想直接控制网络结构的研究者,提升可解释性与调试效率。
我们不直接编程神经网络,而是依赖梯度下降等学习算法从数据中推导网络函数。这种间接方式虽能解决以往难题,但缺乏离散结构,无法将算法直接编译为神经网络——因离散算法通常不可微,与梯度学习不兼容。为此,我们提出 CajaL:一种类型化、高阶且线性的编程语言,旨在探索神经网络的直接编程范式。我们证明 CajaL 程序可编译为线性神经元,使离散算法以可微形式存在,适配梯度学习。通过实现 CajaL,我们在多个实验中将线性神经元与其它神经网络连接,预先定义其部分功能。结果表明,这种结构使网络学习更快速、数据效率更高,且更易调试。核心启示是:线性编程语言为直接编程神经网络提供了路径,促进学习与传统编程结构的深度融合。
原文摘要 · Abstract (English)
We don't program neural networks directly. Instead, we rely on an indirect style where learning algorithms, like gradient descent, determine a neural network's function by learning from data. This indirect style is often a virtue; it empowers us to solve problems that were previously impossible. But it lacks discrete structure. We can't compile most algorithms into a neural network -- even if these algorithms could help the network learn. This limitation occurs because discrete algorithms are not obviously differentiable, making them incompatible with the gradient-based learning algorithms that determine a neural network's function. To address this, we introduce $\textsf{Cajal}$: a typed, higher-order and linear programming language intended to be a minimal vehicle for exploring a direct style of programming neural networks. We prove $\textsf{Cajal}$ programs compile to linear neurons, allowing discrete algorithms to be expressed in a differentiable form compatible with gradient-based learning. With our implementation of $\textsf{Cajal}$, we conduct several experiments where we link these linear neurons against other neural networks to determine part of their function prior to learning. Linking with these neurons allows networks to learn faster, with greater data-efficiency, and in a way that's easier to debug. A key lesson is that linear programming languages provide a path towards directly programming neural networks, enabling a rich interplay between learning and the discrete structures of ordinary programming.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。