给Transformer加人类工作记忆限制，数据少时更准更像人。
Working Memory Constraints Scaffold Learning in Transformers under Data Scarcity
- 用固定窗口和时间衰减机制模拟人类记忆限制
- 10M数据下语法准确率显著提升，且更贴近人类阅读时间
- 适合研究小样本语言学习与认知启发模型
我们研究了将类人工作记忆约束引入Transformer架构，并实现了多种认知启发的注意力变体，包括基于固定宽度窗口和基于时间衰减的注意力机制。在1000万和1亿词的发育可接受数据集上从头训练改进后的GPT-2模型。通过语法判断任务（BLiMP）和与人类阅读时间数据的对齐性评估性能。结果表明，这些认知启发的约束，尤其是固定宽度注意力，在训练数据稀缺时能显著提升语法准确性，并表现出更强的人类处理指标一致性。研究提示此类约束可能作为有益的归纳偏置，引导模型在数据受限条件下形成更稳健的语言表征。
原文摘要 · Abstract (English)
We investigate the integration of human-like working memory constraints into the Transformer architecture and implement several cognitively inspired attention variants, including fixed-width windows based and temporal decay based attention mechanisms. Our modified GPT-2 models are trained from scratch on developmentally plausible datasets (10M and 100M words). Performance is evaluated on grammatical judgment tasks (BLiMP) and alignment with human reading time data. Our results indicate that these cognitively-inspired constraints, particularly fixed-width attention, can significantly improve grammatical accuracy especially when training data is scarce. These constrained models also tend to show a stronger alignment with human processing metrics. The findings suggest that such constraints may serve as a beneficial inductive bias, guiding models towards more robust linguistic representations, especially in data-limited settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。