给Transformer加记忆限制，更贴近人类阅读习惯。
Beyond Static and Linear: What Attention Constraints Best Fit Human Reading Times?
- 引入内容敏感型注意力约束，提升模型对人类阅读时间的预测能力。
- 动态记忆训练下，心理拟合度与语法能力出现分离，表现不一致。
- 适合研究认知建模与自然语言处理交叉的学者参考。
基于Transformer的语言模型广泛用作人类语言处理的模型，但其注意力机制允许无损访问全部前序上下文，与人类有限记忆系统不同。我们假设在注意力机制中加入记忆约束，可提高模型对人类行为数据的拟合度。以往研究多孤立考察单一约束，本文系统比较多种基于注意力的记忆机制，在不同模型规模和训练语料下的表现，评估其对人类阅读时间的预测能力及语法能力。同时对比静态约束（训练全程固定强度）与动态记忆课程。结果发现，对插入词元内容敏感的约束能最好地匹配人类阅读时间，优于基于距离的约束。在动态记忆课程下，心理拟合度与语法能力出现解耦，表明Transformer无法作为统一的认知模型。
原文摘要 · Abstract (English)
Transformer-based language models are widely used as models of human language processing, yet their attention mechanisms allow lossless access to the full preceding context, unlike the limited memory systems of humans. We hypothesize that installing memory constraints into transformers' attention mechanisms can improve their fit to human behavioral data. While previous work has explored individual constraints in isolation, we conduct a systematic comparison of multiple attention-based memory mechanisms across different model sizes and training corpora, evaluating both psychometric predictive power for human reading times and grammatical competence. We additionally compare static constraints, in which the constraint strength is fixed throughout training, to dynamic memory curricula. We find that constraints that are sensitive to the content of intervening tokens consistently achieve the highest alignment with human reading times, outperforming distance-based constraints. We observe a dissociation between psychometric fit and grammatical competence under dynamic memory curricula, suggesting that Transformers cannot serve as a one-size-fits-all cognitive model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。