用动力系统理论把神经网络层转成线性操作，让模型更易理解。
Representing Neural Network Layers as Linear Operations via Koopman Operator Theory
- 将神经网络看作动态系统，用Koopman理论实现非线性层的线性化。
- 在Yin-Yang和MNIST数据集上替换层后，准确率分别降至97.3%和95.8%。
- 时间延迟坐标是有效可观测空间，适合用于线性化网络层。
简单神经网络的强大性能常归因于其非线性激活函数。然而，从线性视角看待神经网络可显著提升其可解释性与可控性。本文借鉴神经网络的动力系统视角，引入柯普曼算子理论及其与动态模态分解（DMD）的联系，构建了一种通过嵌入到合适可观测空间来线性化动态系统的框架。通过将神经网络重新建模为动态系统，我们证明可在预训练多层感知机（MLP）中用有限维线性算子替代非线性层。进一步分析DMD特征值与SVD右奇异向量，发现时间延迟坐标构成一个直接且高效的可观测空间，使柯普曼理论能够有效线性化网络层。我们在Yin-Yang数据集上替换MLP层后，模型准确率达到97.3%，原模型为98.4%；在MNIST数据集上，准确率最高达95.8%，原模型为97.2%。
原文摘要 · Abstract (English)
The strong performance of simple neural networks is often attributed to their nonlinear activations. However, a linear view of neural networks makes understanding and controlling networks much more approachable. We draw from a dynamical systems view of neural networks, offering a fresh perspective by using Koopman operator theory and its connections with dynamic mode decomposition (DMD). Together, they offer a framework for linearizing dynamical systems by embedding the system into an appropriate observable space. By reframing a neural network as a dynamical system, we demonstrate that we can replace the nonlinear layer in a pretrained multi-layer perceptron (MLP) with a finite-dimensional linear operator. In addition, we analyze the eigenvalues of DMD and the right singular vectors of SVD, to present evidence that time-delayed coordinates provide a straightforward and highly effective observable space for Koopman theory to linearize a network layer. Consequently, we replace layers of an MLP trained on the Yin-Yang dataset with predictions from a DMD model, achieving a mdoel accuracy of up to 97.3%, compared to the original 98.4%. In addition, we replace layers in an MLP trained on the MNIST dataset, achieving up to 95.8%, compared to the original 97.2% on the test set.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。