给Transformer加记忆衰减，更贴近人类阅读习惯。
Linear Recency Bias During Training Improves Transformers' Fit to Reading Times
- 在注意力分数中引入时间衰减偏置（ALiBi）
- 模型对人类阅读时间的预测准确率提升
- 不同注意力头的衰减速率差异有助于捕捉语言依赖
近期心理语言学研究将人类阅读时间与语言模型的意外度估计进行对比，探究影响句子处理难度的因素。以往研究显示，Transformer模型的意外度与阅读时间有较强相关性。然而，标准Transformer对全部历史语境无损保留，而人类语言处理模型包含记忆衰减机制。为弥合这一差距，本文评估了一种改进的Transformer模型，该模型采用ALiBi（Press et al., 2022），在注意力分数中加入时间衰减偏置。结果显示，使用ALiBi的意外度估计对人类阅读时间的拟合效果优于标准Transformer基线。进一步分析注意力头发现，ALiBi中各注意力头不同的衰减速率可能帮助模型更好地追踪不同类型的语言依赖关系。
原文摘要 · Abstract (English)
Recent psycholinguistic research has compared human reading times to surprisal estimates from language models to study the factors shaping human sentence processing difficulty. Previous studies have shown a strong fit between surprisal values from Transformers and reading times. However, standard Transformers work with a lossless representation of the entire previous linguistic context, unlike models of human language processing that include memory decay. To bridge this gap, this paper evaluates a modification of the Transformer model that uses ALiBi (Press et al., 2022), a recency bias added to attention scores. Surprisal estimates with ALiBi show an improved fit to human reading times compared to a standard Transformer baseline. A subsequent analysis of attention heads suggests that ALiBi's mixture of slopes -- which determine the rate of memory decay in each attention head -- may play a role in the improvement by helping models with ALiBi to track different kinds of linguistic dependencies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。