发现Transformer在训练中自发形成类似人类情景记忆的时序结构。
Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training
- 用认知科学方法分析GPT-2注意力得分与输出时序模式。
- 观察到类人情景记忆的连续性、首因与近因效应，支持串行回忆倾向。
- 移除诱导头后该效应消失，表明其是时序连续性的关键驱动因素。
我们研究了注意力头和Transformer输出中的上下文时序偏差。采用认知科学方法，分析不同规模的GPT-2模型的注意力得分与输出。在注意力头中，观察到典型的人类情景记忆特征，包括时间连续性、首因效应和近因效应。Transformer输出表现出对上下文序列回忆的倾向。重要的是，当移除诱导头后，这种效应完全消失，而诱导头正是导致连续性效应的主要驱动因素。研究揭示了Transformer在上下文学习过程中如何组织信息的时序结构，为理解其与人类记忆和学习的异同提供了洞见。
原文摘要 · Abstract (English)
We investigate in-context temporal biases in attention heads and transformer outputs. Using cognitive science methodologies, we analyze attention scores and outputs of the GPT-2 models of varying sizes. Across attention heads, we observe effects characteristic of human episodic memory, including temporal contiguity, primacy and recency. Transformer outputs demonstrate a tendency toward in-context serial recall. Importantly, this effect is eliminated after the ablation of the induction heads, which are the driving force behind the contiguity effect. Our findings offer insights into how transformers organize information temporally during in-context learning, shedding light on their similarities and differences with human memory and learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。