数据对称性不总带来守恒量，但特定损失下可催生新守恒量。
Conservation Laws from Data Symmetry in Neural Networks
- 用张量化网络建模参数与输入的分离依赖关系
- 证明多数情况下数据对称性不产生额外守恒量
- 均方误差下数据增强可引出新守恒量，适合理论研究者
我们研究训练数据的内在对称性是否在神经网络的梯度流训练中导致守恒量。假设损失函数为解析且非多项式，我们证明数据对称性通常不会引发任何额外的运动积分。然而，对于均方误差（MSE）损失，在某些情形下数据增强会带来额外的守恒量。为此，我们构建了一个框架，利用张量化网络（tensorizable networks）来描述该现象。张量化网络是一类架构，其参数与输入的依赖关系可通过中间表示分离。这类网络包括线性网络、多项式网络以及Lightning Attention。
原文摘要 · Abstract (English)
We explore whether intrinsic symmetries of the training data lead to conserved quantities during gradient-flow training of neural networks. Under the assumption that the loss function is analytic and non-polynomial, we prove that data symmetries generically do not induce any additional integrals of motion. For mean squared error (MSE) loss, on the other hand, there are situations in which data augmentation yields extra conserved quantities. We build a framework, utilizing \emph{tensorizable networks} to describe this phenomenon. Tensorizable networks are a family of architectures whose dependence on parameters and inputs can be separated using an intermediate representation. They include linear and polynomial networks, as well as Lightning Attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。