冻结记忆模块可显著提升RNN在长序列上的泛化能力
Exploring Learnability in Memory-Augmented Recurrent Neural Networks: Precision, Stability, and Empirical Insights
- 通过冻结记忆组件稳定时序依赖，改善模型学习
- 在Penn Treebank上测试困惑度降至120.5，优于原模型
- 适合研究长序列建模与稳定记忆设计的学者
本研究探讨无记忆与带记忆RNN的可学习性，二者在理论上等价于下推自动机。实验表明，这些模型在长序列上常难以泛化，更依赖精度而非掌握符号语法。对全训练与组件冻结模型的对比显示，冻结记忆组件显著提升性能，在Penn Treebank数据集上测试困惑度从123.5降至120.5。记忆冻结模型在长序列上保留高达90%的初始性能，而标准模型下降至60%。理论分析指出，冻结记忆能稳定时序依赖，实现稳健收敛。研究强调需采用稳定记忆设计与长序列评估以揭示RNN真实学习极限。
原文摘要 · Abstract (English)
This study explores the learnability of memory-less and memory-augmented RNNs, which are theoretically equivalent to Pushdown Automata. Empirical results show that these models often fail to generalize on longer sequences, relying more on precision than mastering symbolic grammar. Experiments on fully trained and component-frozen models reveal that freezing the memory component significantly improves performance, achieving state-of-the-art results on the Penn Treebank dataset (test perplexity reduced from 123.5 to 120.5). Models with frozen memory retained up to 90% of initial performance on longer sequences, compared to a 60% drop in standard models. Theoretical analysis suggests that freezing memory stabilizes temporal dependencies, leading to robust convergence. These findings stress the need for stable memory designs and long-sequence evaluations to understand RNNs true learnability limits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。