用句法结构建模记忆检索，发现注意力是通用提取机制
If Attention Serves as a Cognitive Model of Human Memory Retrieval, What is the Plausible Memory Representation?
- 用句法结构作为表示单元的Transformer语法模型
- 注意力熵预测阅读时间表现优于传统模型
- 适合关注语言认知与神经科学交叉的研究者
计算心理语言学研究表明，注意力机制与人类记忆检索存在有趣相似性，但现有研究多聚焦于基于词元表示的普通Transformer。然而，计算心理语言学也证实，句法结构能更好解释人类句子处理，而词元层面因素无法完全涵盖。本文探讨了仅以句法结构为表示单元的Transformer语法（TG）的注意力机制是否可作为人类记忆检索的认知模型，并采用归一化注意力熵（NAE）作为模型与人类之间的连接假设。实验表明，相较于普通Transformer，TG在预测自控阅读时间方面具有更优的预测能力，进一步分析揭示两模型均有独立贡献。结果表明，人类句子处理涉及双重记忆表征——一种基于句法结构，另一种基于词元序列——而注意力是通用的记忆检索算法，强调将句法结构纳入表示单元的重要性。
原文摘要 · Abstract (English)
Recent work in computational psycholinguistics has revealed intriguing parallels between attention mechanisms and human memory retrieval, focusing primarily on vanilla Transformers that operate on token-level representations. However, computational psycholinguistic research has also established that syntactic structures provide compelling explanations for human sentence processing that token-level factors cannot fully account for. In this paper, we investigate whether the attention mechanism of Transformer Grammar (TG), which uniquely operates on syntactic structures as representational units, can serve as a cognitive model of human memory retrieval, using Normalized Attention Entropy (NAE) as a linking hypothesis between models and humans. Our experiments demonstrate that TG's attention achieves superior predictive power for self-paced reading times compared to vanilla Transformer's, with further analyses revealing independent contributions from both models. These findings suggest that human sentence processing involves dual memory representations -- one based on syntactic structures and another on token sequences -- with attention serving as the general memory retrieval algorithm, while highlighting the importance of incorporating syntactic structures as representational units.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。