用动态权重模拟短期记忆，连接人工智能与脑科学。
Fast weight programming and linear transformers: from machine learning to neurobiology
- 用二维矩阵状态替代传统向量，实现快速权重编程。
- 可作为短时记忆机制，支持高效语言建模与计算。
- 为神经网络与大脑突触可塑性提供统一理解框架。
近期机器学习，特别是语言建模领域的进展，催生了一类新型循环神经网络（RNN）架构：与传统使用向量形式隐藏状态的RNN不同，这类网络采用二维矩阵形式的隐藏状态。这种2D状态的RNN被称为快速权重编程器（Fast Weight Programmers, FWPs），可被解释为一种神经网络，其突触权重（称作快速权重）随输入动态变化，作为短期记忆存储；这些权重的修改由另一个网络（称为程序员）控制，其参数通过梯度下降等方法训练。本文综述了FWP的技术基础、计算特性，并探讨其与变换器（transformers）和状态空间模型的联系。同时讨论了其与大脑突触可塑性模型的关联，提示人工智能与自然智能可能走向融合。
原文摘要 · Abstract (English)
Recent advances in artificial neural networks for machine learning, and language modeling in particular, have established a family of recurrent neural network (RNN) architectures that, unlike conventional RNNs with vector-form hidden states, use two-dimensional (2D) matrix-form hidden states. Such 2D-state RNNs, known as Fast Weight Programmers (FWPs), can be interpreted as a neural network whose synaptic weights (called fast weights) dynamically change over time as a function of input observations, and serve as short-term memory storage; corresponding synaptic weight modifications are controlled or programmed by another network (the programmer) whose parameters are trained (e.g., by gradient descent). In this Primer, we review the technical foundations of FWPs, their computational characteristics, and their connections to transformers and state space models. We also discuss connections between FWPs and models of synaptic plasticity in the brain, suggesting a convergence of natural and artificial intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。