用权重空间建模序列,实现无需梯度的实时自适应学习。
Weight-Space Linear Recurrent Neural Networks
- 将隐藏状态设为独立网络的权重和偏置,通过输入差值驱动更新
- 在6个真实数据集上4个进入前三,物理先验版本性能超基准10倍以上
- 适合需要快速适应新任务或融合领域知识的场景
我们提出WARP(Weight-space Adaptive Recurrent Prediction),一种将权重空间学习与线性递归结合的简单而强大的模型,重新定义序列建模。不同于传统RNN将时序动态压缩为固定维数的隐藏状态,WARP显式地将隐藏状态参数化为一个独立辅助神经网络的权重和偏置,并利用输入差异驱动其递归更新。这一类脑设计使得模型可在测试阶段实现高效的无梯度自适应,具备上下文学习能力,并能无缝融入特定领域的物理先验。实证表明,WARP在多个分类任务中表现达到或超越现有最优基线,在6个真实世界挑战性数据集中有4个位列前三。此外,在序列图像补全、多变量时间序列预测和动力系统重构等任务中,充分展示了其表达力与泛化能力。值得注意的是,引入物理先验的变体模型性能比次优模型高出超过10倍。消融实验验证了关键组件的必要性,确立了权重空间线性RNN作为自适应机器智能的变革范式。
原文摘要 · Abstract (English)
We introduce WARP (Weight-space Adaptive Recurrent Prediction), a simple yet powerful model that unifies weight-space learning with linear recurrence to redefine sequence modeling. Unlike conventional recurrent neural networks (RNNs) which collapse temporal dynamics into fixed-dimensional hidden states, WARP explicitly parametrizes its hidden state as the weights and biases of a distinct auxiliary neural network, and uses input differences to drive its recurrence. This brain-inspired formulation enables efficient gradient-free adaptation of the auxiliary network at test-time, in-context learning abilities, and seamless integration of domain-specific physical priors. Empirical validation shows that WARP matches or surpasses state-of-the-art baselines on diverse classification tasks, featuring in the top three in 4 out of 6 real-world challenging datasets. Furthermore, extensive experiments across sequential image completion, multivariate time series forecasting, and dynamical system reconstruction demonstrate its expressiveness and generalisation capabilities. Remarkably, a physics-informed variant of our model outperforms the next best model by more than 10x. Ablation studies confirm the architectural necessity of key components, solidifying weight-space linear RNNs as a transformative paradigm for adaptive machine intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。