RNN在模加任务中用傅里叶稀疏结构实现高效计算，揭示了低秩权重的数学本质。
Low Rank and Sparse Fourier Structure in Recurrent Networks Trained on Modular Addition
- RNN通过学习傅里叶乘法机制解决模加问题，权重具低秩特性。
- 模型在傅里叶空间呈现稀疏表示，仅依赖少数关键频率分量。
- 移除单个频率影响小，但多频段移除会导致性能骤降，体现鲁棒性与敏感性平衡。
模加任务是观察深度学习中经验现象（如“领悟”现象）的有力测试平台。先前研究发现，单层Transformer架构通过学习傅里叶乘法电路来解决模加任务。本文表明，训练于模加任务的循环神经网络（RNN）同样采用傅里叶乘法策略。我们识别出模型权重中的低秩结构，并将模型组件归因于特定傅里叶频率，从而在傅里叶空间形成稀疏表示。实验显示，该RNN对单个频率的移除具有鲁棒性，但随着更多频率被删除，性能急剧下降。
原文摘要 · Abstract (English)
Modular addition tasks serve as a useful test bed for observing empirical phenomena in deep learning, including the phenomenon of \emph{grokking}. Prior work has shown that one-layer transformer architectures learn Fourier Multiplication circuits to solve modular addition tasks. In this paper, we show that Recurrent Neural Networks (RNNs) trained on modular addition tasks also use a Fourier Multiplication strategy. We identify low rank structures in the model weights, and attribute model components to specific Fourier frequencies, resulting in a sparse representation in the Fourier space. We also show empirically that the RNN is robust to removing individual frequencies, while the performance degrades drastically as more frequencies are ablated from the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。