用符号回归提升神经网络可解释性，实现稀疏数据下的系统建模。
T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoders
- 用Transformer+符号回归替代传统RNN，增强时间建模能力。
- 在低数据到高数据场景下均能准确预测混沌系统动态。
- 模型可直接生成可读方程，适合需要解释性的科研场景。
SHallow REcurrent Decoders（SHRED）在从稀疏传感器数据中进行系统识别和预测方面表现优异，具有轻量、计算高效的特点，可在消费级笔记本上训练。其结构由循环神经网络（RNN）用于时间编码，简单多层感知机（MLP）用于空间解码。尽管结构简单，SHRED仍能直接从稀疏数据中预测不同物理、空间和时间尺度下的混沌动力系统。本文提出T-SHRED，在时间编码部分引入基于符号回归的Transformer架构，并嵌入稀疏非线性动力学识别（SINDy）注意力机制，对潜在空间施加稀疏正则化，避免自回归长期预测，同时实现即时符号化解释。符号回归在训练过程中学习并正则化潜在空间的动力学特性，显著提升模型可解释性。我们在三个不同动力系统上评估了T-SHRED性能，覆盖从低数据到高数据的多种情形。
原文摘要 · Abstract (English)
SHallow REcurrent Decoders (SHRED) are effective for system identification and forecasting from sparse sensor measurements. Such models are light-weight and computationally efficient, allowing them to be trained on consumer laptops. SHRED-based models rely on Recurrent Neural Networks (RNNs) and a simple Multi-Layer Perceptron (MLP) for the temporal encoding and spatial decoding respectively. Despite the relatively simple structure of SHRED, they are able to predict chaotic dynamical systems on different physical, spatial, and temporal scales directly from a sparse set of sensor measurements. In this work, we modify SHRED by leveraging transformers (T-SHRED) embedded with symbolic regression for the temporal encoding, circumventing auto-regressive long-term forecasting for physical data. This is achieved through a new sparse identification of nonlinear dynamics (SINDy) attention mechanism into T-SHRED to impose sparsity regularization on the latent space, which also allows for immediate symbolic interpretation. Symbolic regression improves model interpretability by learning and regularizing the dynamics of the latent space during training. We analyze the performance of T-SHRED on three different dynamical systems ranging from low-data to high-data regimes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。